Video delivery method for delivering videos captured from a plurality of viewpoints, video reception method, server, and terminal device

ABSTRACT

There is provided a video delivery method for a server that delivers to a terminal device a video signal based on a plurality of videos obtained by shooting an identical scene from a plurality of different viewpoints over a plurality of successive periods, the method including: selecting a selected video from among the plurality of videos for each of the plurality of periods in accordance with rating values of the plurality of videos, the rating values being given by a plurality of users; and transmitting to the terminal device the video signal for successively replaying a plurality of the selected videos respectively selected for the plurality of periods.

BACKGROUND

1. Technical Field

The present disclosure relates to a video delivery method for deliveringvideos captured from a plurality of viewpoints, a video receptionmethod, a server, and a terminal device.

2. Description of the Related Art

As a video delivery method, a technique described in Japanese UnexaminedPatent Application Publication No. 2009-206625 is available, forexample. Further, a video delivery method for delivering a videocaptured from a plurality of viewpoints is available (for example, seeJapanese Unexamined Patent Application Publication No. 2012-094990). Inthese video delivery methods, a user is able to specify and view anyvideo from among a plurality of videos obtained by shooting a specificscene from different viewpoints.

It is desired that the video delivery methods enable delivery of asuitable video on the basis of a plurality of videos.

SUMMARY

One non-limiting and exemplary embodiment provides a video deliverymethod, a video reception method, a server, or a terminal device withwhich it is possible to deliver a suitable video on the basis of aplurality of videos.

In one general aspect, the techniques disclosed here feature a methodcomprising: obtaining a plurality of videos by taking an identical scenefrom a plurality of different viewpoints over a plurality of successiveperiods; selecting a first selected video from among the plurality ofvideos for each of the plurality of periods to select a plurality offirst video selected videos in accordance with rating values of theplurality of videos, the rating values being given by a plurality ofusers; and transmitting from a server to a terminal device, a firstvideo signal for successively replaying the plurality of first selectedvideos respectively selected for the plurality of periods.

The present disclosure is able to provide a video delivery method, avideo reception method, a server, or a terminal device with which it ispossible to deliver a suitable video on the basis of a plurality ofvideos.

It should be noted that general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a storage medium, or any selective combination thereof.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of a video deliverysystem according to a first embodiment;

FIG. 2 is a block diagram of a terminal device according to the firstembodiment;

FIG. 3 is a flowchart illustrating a process performed by the terminaldevice according to the first embodiment;

FIG. 4 is a flowchart illustrating a viewpoint video switching processperformed by the terminal device according to the first embodiment;

FIG. 5 is a flowchart illustrating a rating giving process performed bythe terminal device according to the first embodiment;

FIG. 6 is a flowchart illustrating an information obtaining processperformed by the terminal device according to the first embodiment;

FIG. 7 is a block diagram of a server according to the first embodiment;

FIG. 8 is a flowchart illustrating a process performed by the serveraccording to the first embodiment;

FIG. 9 is a flowchart illustrating a viewpoint video switching processperformed by the server according to the first embodiment;

FIG. 10 is a flowchart illustrating a rating giving process performed bythe server according to the first embodiment;

FIG. 11 is a flowchart illustrating an information obtaining processperformed by the server according to the first embodiment;

FIG. 12 is a diagram illustrating an example of an operation in afavorite subject switching mode according to the first embodiment;

FIG. 13 is a diagram illustrating an example of an operation in arecommended viewpoint video switching mode according to the firstembodiment;

FIG. 14 is a diagram illustrating an example of an operation in arecommended subject switching mode according to the first embodiment;

FIG. 15 is a diagram illustrating an example of a display screenaccording to the first embodiment;

FIG. 16 is a diagram illustrating an example of a display screenaccording to the first embodiment;

FIG. 17 is a diagram illustrating an example of a display screenaccording to the first embodiment;

FIG. 18 is a flowchart of a video delivery method performed by theserver according to the first embodiment;

FIG. 19 is a flowchart of a video reception method performed by theterminal device according to the first embodiment;

FIG. 20 is a diagram illustrating a configuration of a video informationprocessing system;

FIG. 21 is a diagram illustrating an example of a notification screendisplayed upon start-up of a camera;

FIG. 22 is a diagram illustrating an overall configuration of a contentsupply system that implements a content delivery service;

FIG. 23 is a diagram illustrating an overall configuration of a digitalbroadcasting system;

FIG. 24 is a diagram illustrating an example of a smartphone; and

FIG. 25 is a block diagram illustrating an example of a configuration ofthe smartphone.

DETAILED DESCRIPTION

A video delivery method according to an aspect of the present disclosureis a video delivery method for a server that delivers to a terminaldevice a first video signal based on a plurality of videos obtained byshooting an identical scene from a plurality of different viewpointsover a plurality of successive periods, the method including: selectinga first selected video from among the plurality of videos for each ofthe plurality of periods in accordance with rating values of theplurality of videos, the rating values being given by a plurality ofusers; and transmitting to the terminal device the first video signalfor successively replaying a plurality of first selected videosrespectively selected for the plurality of periods.

Accordingly, with the video delivery method, it is possible to deliver asuitable video based on ratings given by a plurality of users.

For example, each of the rating values may be a rating value of each ofthe plurality of videos, and in the selecting of the first selectedvideo, for each period among the plurality of periods, a video having ahighest rating value or having a rating value higher than apredetermined value in the period may be selected from among theplurality of videos as the first selected video.

Accordingly, with the video delivery method, it is possible to deliver avideo for successively replaying highly rated viewpoint videos on thebasis of ratings given by a plurality of users.

For example, each of the rating values may be a rating value of each ofa plurality of subjects included in the plurality of videos, and in theselecting of the first selected video, for each of the plurality ofperiods, a video in which a recommended subject having a highest ratingvalue or having a rating value higher than a predetermined value isvisible may be selected from among the plurality of videos as the firstselected video.

Accordingly, with the video delivery method, it is possible to deliver avideo for successively replaying viewpoint videos in which a highlyrated subject is visible on the basis of ratings given by a plurality ofusers.

For example, in the selecting of the first selected video, for each ofthe plurality of periods, a video in which the recommended subject has alargest area may be selected from among the plurality of videos as thefirst selected video.

Accordingly, with the video delivery method, it is possible to deliver avideo for successively replaying viewpoint videos in which a highlyrated subject is visible in a close-up manner on the basis of ratingsgiven by a plurality of users.

For example, the video delivery method may include, in a case where thefirst video signal has an amount of data larger than an upper limitbased on a reception band of the terminal device, selecting a secondselected video from among the plurality of videos for each of theplurality of periods in accordance with the rating values of theplurality of videos; and transmitting to the terminal device a secondvideo signal for successively replaying a plurality of second selectedvideos respectively selected for the plurality of periods. At least oneof the plurality of second selected videos respectively selected for theplurality of periods may be different from the plurality of firstselected videos respectively selected for the plurality of periods, andthe second video signal may have an amount of data smaller than theupper limit.

Accordingly, with the video delivery method, it is possible to deliver avideo signal having a small amount of data in a case where the receptionband of the terminal device is narrow.

For example, the plurality of videos may be transmitted from a pluralityof cameras to the server in real time, and in a case where transmissionof a video corresponding to the first selected video from acorresponding one of the cameras is discontinued, in the selecting ofthe first selected video, a second selected video different from thefirst selected video may be selected, and in the transmitting of thefirst video signal, a second video signal for replaying the secondselected video may be transmitted to the terminal device.

Accordingly, with the video delivery method, it is possible to suppressdiscontinuous video delivery in a case where transmission of a videofrom a camera is discontinued.

For example, each of the rating values may be a rating value of each ofthe plurality of videos, and in the selecting of the first selectedvideo, for each period among the plurality of periods, a video having arating value lower than a predetermined value in the period may beselected from among the plurality of videos as the first selected video.

Accordingly, with the video delivery method, it is possible to reduceblind spots that are created in a case of use in surveillance.

For example, in the transmitting of the first video signal, the firstvideo signal that is generated by connecting together the plurality offirst selected videos respectively selected for the plurality of periodsmay be transmitted to the terminal device.

For example, in the transmitting of the first video signal, theplurality of videos and information for identifying the plurality offirst selected videos respectively selected for the plurality of periodsmay be transmitted to the terminal device as the first video signal.

Further, a video reception method according to an aspect of the presentdisclosure is a video reception method for a terminal device thatreceives from a server a video signal based on a plurality of videosobtained by shooting an identical scene from a plurality of differentviewpoints over a plurality of successive periods, the method including:receiving from the server the video signal for successively replaying aplurality of selected videos respectively selected for the plurality ofperiods from among the plurality of videos in accordance with ratingvalues of the plurality of videos, the rating values being given by aplurality of users; and successively replaying the plurality of selectedvideos in accordance with the video signal.

Accordingly, with the video reception method, it is possible to replay asuitable video based on ratings given by a plurality of users.

For example, each of the rating values may be a rating value of each ofthe plurality of videos, and each of the selected videos may be a videothat is selected from among the plurality of videos for each periodamong the plurality of periods and that has a highest rating value or arating value higher than a predetermined value in the period.

Accordingly, with the video reception method, it is possible tosuccessively replay highly rated viewpoint videos on the basis ofratings given by a plurality of users.

For example, each of the rating values may be a rating value of each ofa plurality of subjects included in the plurality of videos, and each ofthe selected videos may be a video which is selected from among theplurality of videos for each of the plurality of periods and in which asubject having a highest rating value or having a rating value higherthan a predetermined value is visible.

Accordingly, with the video reception method, it is possible tosuccessively replay viewpoint videos in which a highly rated subject isvisible on the basis of ratings given by a plurality of users.

For example, the video reception method may further include displayingany video among the plurality of videos. In the displaying of the video,a subject having a highest rating value or having a rating value higherthan a predetermined value may be highlighted and displayed among aplurality of subjects included in the displayed video.

Accordingly, with the video reception method, it is possible tocommunicate a highly rated subject to a user.

For example, the video reception method may further include displayingany video among the plurality of videos; and displaying, in a case whereany subject is selected by a user from among a plurality of subjectsincluded in the displayed video, a menu for rating the selected subject.

Accordingly, with the video reception method, it is possible to increasethe user's convenience.

For example, the video reception method may further include displayingany video among the plurality of videos; and displaying, in a case whereany subject is selected by a user from among a plurality of subjectsincluded in the displayed video, a list of videos for the plurality ofperiods among the plurality of videos, the videos in the list includingthe selected subject and having rating values higher than apredetermined value.

Accordingly, with the video reception method, it is possible to increasethe user's convenience.

For example, the video reception method may further include displayingany video among the plurality of videos; and displaying, in a case whereany subject is selected by a user from among a plurality of subjectsincluded in the displayed video, a tracking video that is a videogenerated by connecting together the plurality of videos for trackingthe selected subject.

Accordingly, with the video reception method, it is possible to increasethe user's convenience.

For example, the video reception method may further include displayingan output screen that includes a first screen on which any of theplurality of videos is displayed, and a second screen which is smallerthan the first screen and on which the plurality of selected videos thatare replayed are displayed; and displaying, in a case where the secondscreen is selected by a user, the plurality of selected videos on thefirst screen.

Accordingly, with the video reception method, it is possible to increasethe user's convenience.

For example, each of the rating values may be a rating value of each ofthe plurality of videos, and each of the selected videos may be a videothat is selected from among the plurality of videos for each periodamong the plurality of periods and that has a rating value lower than apredetermined value in the period.

Further, a server according to an aspect of the present disclosure is aserver for delivering to a terminal device a video signal based on aplurality of videos obtained by shooting an identical scene from aplurality of different viewpoints over a plurality of successiveperiods, the server including a processor, and a memory having storedtherein instructions which, when executed by the processor, cause theprocessor to perform operations including: selecting a selected videofrom among the plurality of videos for each of the plurality of periodsin accordance with rating values of the plurality of videos, the ratingvalues being given by a plurality of users; and transmitting to theterminal device the video signal for successively replaying a pluralityof selected videos respectively selected for the plurality of periods.

Accordingly, with the server, it is possible to deliver a suitable videobased on ratings given by a plurality of users.

Further, a terminal device according to an aspect of the presentdisclosure is a terminal device for receiving from a server a videosignal based on a plurality of videos obtained by shooting an identicalscene from a plurality of different viewpoints over a plurality ofsuccessive periods, the terminal device including a processor, and amemory having stored therein instructions which, when executed by theprocessor, cause the processor to perform operations including:receiving from the server the video signal for successively replaying aplurality of selected videos respectively selected for the plurality ofperiods from among the plurality of videos in accordance with ratingvalues of the plurality of videos, the rating values being given by aplurality of users; and successively replaying the plurality of selectedvideos in accordance with the video signal.

Accordingly, with the terminal device, it is possible to replay asuitable video based on ratings given by a plurality of users.

It should be noted that the general or specific aspects may beimplemented as a system, a method, an integrated circuit, a computerprogram, a storage medium, such as a computer-readable compact discread-only memory, or any selective combination thereof.

Hereinafter, embodiments will be specifically described with referenceto the drawings. Note that any of the embodiments described below is anoperative example of the present disclosure. Numerical values, forms,materials, constituent elements, arrangements and connections ofconstituent elements, steps, the order of steps, and so on described inthe following embodiments are merely examples and are not intended tolimit the present disclosure. Further, among the constituent elementsdescribed in the following embodiments, a constituent element notdescribed in an independent claim that defines the most generic conceptwill be described as a preferred constituent element.

First Embodiment

In a video delivery system according to this embodiment, a plurality ofselected videos are selected from among a plurality of viewpoint videoson the basis of ratings given by a plurality of users for each period,and video data obtained by connecting the selected selected videostogether is delivered to a terminal device.

First, a configuration of a video delivery system 100 according to thisembodiment is described. FIG. 1 is a block diagram illustrating anoverall configuration of the video delivery system 100 according to thisembodiment. The video delivery system 100 includes a plurality ofcameras 101, a terminal device 102, and a server 103 that can eachcommunicate over a network 104A or 104B.

The plurality of cameras 101 generate a plurality of viewpoint videos byshooting the same scene over the same time period from differentviewpoints. The cameras 101 are carried by a plurality of usersrespectively. For example, the plurality of cameras 101 are owned by aplurality of spectators who are in a place, such as in a sport stadium.The plurality of viewpoint videos captured by the plurality of cameras101 are transmitted to the server 103 over the network 104A. The cameras101 also obtain data, such as position information, used to calculatethe degree of relation among the viewpoint videos and transmit the datato the server 103. The camera 101 may be any device as long as thedevice has at least an image capturing function, and is a digital stillcamera, a digital video camera, a smartphone, or a portable terminal,for example. Examples of the plurality of cameras 101 may include notonly cameras owned by spectators but also fixed cameras or cameras forbroadcasting.

The networks 104A and 104B correspond to the Internet, for example.Although the networks 104A and 104B are separately illustrated in FIG.1, the plurality of cameras 101, the terminal device 102, and the server103 may be connected to one another over a single network.Alternatively, part or all of communication among the apparatuses may bedirectly performed without the network 104A or 104B. Further,connections among the apparatuses may be wired connections or wirelessconnections.

The terminal device 102 is a terminal used by a user 106 and isconnected to a monitor 105 on which a video is displayed. For example,the terminal device 102 is a smartphone, a portable terminal, or apersonal computer (PC). Note that the terminal device 102 may have afunction similar to that of the camera 101, the user 106 may be aspectator described above, and the user 106 may view a video in a placeother than a stadium described above.

The terminal device 102 receives a viewpoint video from the server 103in accordance with an instruction provided by the user 106 and outputsthe viewpoint video to the monitor 105 by using a method based on theinstruction provided by the user 106. Note that the monitor 105 may beany output device, such as a PC monitor, a tablet terminal, asmartphone, a portable phone, or a monitor of a note PC. Although theterminal device 102 and the monitor 105 are separately illustrated here,the monitor 105 may be included in the terminal device 102.

An instruction from the user 106 is provided through a screen touchoperation or by using a mouse or the like, for example; however, anyinput device may be used to provide an instruction. Instructionsprovided by the user 106 include instructions for selecting a viewpointvideo to be viewed, making a comment on a viewpoint video, rating aviewpoint video, making a comment on a subject that is visible in aviewpoint video, rating a subject, checking information about a subject,placing an order for an item related to a subject, and the like. Some ofthe instructions provided by the user 106 are transmitted to the server103.

The server 103 temporarily retains a plurality of viewpoint videostransmitted from the plurality of cameras 101. The server 103 transmitssome or all of the plurality of retained viewpoint videos to theterminal device 102 in response to a request from the terminal device102. The server 103 performs a process, such as switching betweenviewpoint videos, in response to a request from the terminal device 102and transmits video data obtained in the process to the terminal device102. The server 103 receives from the terminal device 102 ratings onviewpoint videos or subjects in viewpoint videos given by users,summarizes the ratings as comments and ratings on the viewpoint videosor the subjects, and retains the ratings and comments in a database.

The server 103 transmits information to the terminal device 102 inresponse to a request from the terminal device 102 for presentinginformation about a viewpoint video or a subject. The server 103identifies a subject included in a plurality of viewpoint videoscaptured by the cameras 101 by using information about the subjectspecified by the user 106, the information being transmitted from theterminal device 102. Specifically, the server 103 receives subjectinformation (name, uniform number, or the like) about a subjectspecified by the user 106 or pixel information about a subject in aviewpoint video specified by the user 106 through a touch operation, amouse operation, a gesture operation, a line-of-sight operation, or avoice operation and identifies the subject by performing imageprocessing, such as feature value matching of the plurality of viewpointvideos captured by the cameras 101 with the pixel information that hasbeen received. Note that the server 103 may create a combined image bycombining the plurality of viewpoint videos captured by the plurality ofcameras 101. This combined image may be included in a plurality ofviewpoint videos that are transmitted to the terminal device 102.

Note that a plurality of viewpoint videos may be transmitted from theplurality of cameras 101 in real time, and the user 106 may view theviewpoint videos in real time by using the terminal device 102.Alternatively, at least one of transmission and viewing of videos neednot be performed in real time. That is, the user 106 may view aviewpoint video captured in the past at any time. Transmission andreception of video signals (videos) mainly means streaming transmissionand reception in which video signals are successively transmitted orreceived.

Hereinafter, a configuration of each apparatus is described. FIG. 2 is ablock diagram illustrating a configuration of the terminal device 102.The terminal device 102 includes a reception unit 111, an accumulationunit 112, an image processing unit 113, an output unit 114, atransmission unit 115, and a control unit 116.

The reception unit 111 receives video data transmitted from the server103. Specifically, the video data includes a viewpoint video specifiedby the user 106. The video data also includes a video that is closelyrelated to the viewpoint video as sub-data. For example, the video isanother viewpoint video captured at a position closer to that of aviewpoint video that is being viewed, another viewpoint video includinga subject which is visible in a viewpoint video that is being viewed, orthe like. In this way, the server 103 may simultaneously transmit to theterminal device 102 a video that is closely related to a viewpoint videothat is being viewed together with the video that is being viewed. Thevideo data includes not only a viewpoint video specified by the user 106but also comments or ratings on the viewpoint video or a subject,related information, and the like. The server 103 transmits informationand the like about a viewpoint video that is determined to be mostcommonly viewed as a result of summarization by the server 103.

The accumulation unit 112 temporarily retains the video data received bythe reception unit 111, which enables seamless display upon switchingbetween viewpoints.

The image processing unit 113 performs a decoding process in a casewhere a viewpoint video has been coded. The image processing unit 113generates an output video to be presented to the user 106 by using aviewpoint video, information about the viewpoint video, and informationabout a subject, the information being received from the server 103. Inthis case, the image processing unit 113 may separately obtain from theserver 103 statistical information about viewpoint videos that arecommonly viewed and about subjects on which a large number of commentsare made or which are highly rated and the like and create an outputvideo in which the information about the viewpoints and subjects arehighlighted. For example, the image processing unit 113 may illuminatean icon for selecting a viewpoint video that is commonly viewed orilluminate the outline or the like of a subject that is highly rated tothereby highlight the viewpoint video or the subject.

The output unit 114 outputs the output video generated by the imageprocessing unit 113 to the monitor 105.

The control unit 116 accepts a user instruction. Here, a userinstruction includes (1) a viewpoint video switching request, (2) arating giving request, and (3) an information obtaining request.

The viewpoint video switching request (1) is a request for switchingbetween viewpoint videos for viewing and has (1-1) a favorite viewpointvideo switching mode, (1-2) a favorite subject switching mode, (1-3) arecommended viewpoint video switching mode, and (1-4) a recommendedsubject switching mode.

The favorite viewpoint video switching mode (1-1) is a mode forselecting a favorite viewpoint video registered in advance by the user106. The favorite subject switching mode (1-2) is a mode for selecting aviewpoint video that includes a favorite subject registered in advanceby the user 106. In the favorite viewpoint video switching mode and inthe favorite subject switching mode, the user 106 can select a viewpointvideo or a subject at any time while viewing a video as well as afavorite viewpoint video or a viewpoint video including a favoritesubject registered in advance by the user 106.

The recommended viewpoint video switching mode (1-3) is a mode forselecting a viewpoint video highly rated by a plurality of users. Therecommended subject switching mode (1-4) is a mode for selecting aviewpoint video that includes a subject highly rated by a plurality ofusers.

The rating giving request (2) is a request for rating a viewpoint videoor a subject and has (2-1) a viewpoint video rating giving mode and(2-2) a subject rating giving mode. The viewpoint video rating givingmode (2-1) is a mode for the user 106 to rate a viewpoint video. Thesubject rating giving mode (2-2) is a mode for the user 106 to rate asubject included in a viewpoint video.

The information obtaining request (3) is a request for obtaininginformation about a viewpoint video or a subject and has (3-1) aviewpoint video information obtaining mode and (3-2) a subjectinformation obtaining mode. The viewpoint video information obtainingmode (3-1) is a mode for obtaining information about a viewpoint video.The subject information obtaining mode (3-2) is a mode for obtaininginformation about a subject included in a viewpoint video.

In a case where a user instruction is the viewpoint video switchingrequest (1), the control unit 116 transmits information about aspecified viewpoint video position or information about a specifiedsubject position to the server 103 via the transmission unit 115.

In a case where a user instruction is the rating giving request (2), thecontrol unit 116 transmits information about a comment or a rating on aviewpoint video and time information about the time when the rating wasgiven, information about a comment or a rating on a specified subjectand time information about the time when the rating was given, or thelike to the server 103 via the transmission unit 115.

In a case where a user instruction is the information obtaining request(3), the control unit 116 transmits information about a specifiedviewpoint video position or information about a specified subjectposition, information about information content to be obtained, and thelike to the server 103 via the transmission unit 115. Here, informationcontent to be obtained is information about a viewpoint video or asubject, and examples thereof include (1) comments or ratings on theviewpoint video or the subject, (2) a profile of the videographer of theviewpoint video or that of the subject, and (3) information about othercontent created by the videographer of the viewpoint video orinformation about items related to the subject.

In a case where a subject is specified, that is, in a case where a userinstruction is a request having the favorite subject switching mode(1-2), the subject rating giving mode (2-2) or the subject informationobtaining mode (3-2), subject information (name, uniform number, or thelike) about the subject specified by the user 106 or pixel informationabout the subject in a viewpoint video specified by the user 106 througha touch operation, a mouse operation, a gesture operation, aline-of-sight operation, or a voice operation is transmitted to theserver 103 in order for the server 103 to identify the subject specifiedby the user 106.

Hereinafter, an operation of the terminal device 102 is described. FIG.3 is a flowchart of the entire process performed by the terminal device102.

First, the terminal device 102 determines whether a user instruction isthe viewpoint video switching request (1), the rating giving request(2), or the information obtaining request (3) (steps S101, S103, andS105).

If the user instruction is the viewpoint video switching request (Yes instep S101), the terminal device 102 performs a viewpoint video switchingprocess (step S102). If the user instruction is the rating givingrequest (Yes in step S103), the terminal device 102 performs a ratinggiving process (step S104). If the user instruction is the informationobtaining request (Yes in step S105), the terminal device 102 performsan information obtaining process (step S106).

Note that the terminal device 102 need not perform all of the steps fordetermination described above. The terminal device 102 may onlydetermine whether the user instruction is the viewpoint video switchingrequest, for example, that is, may optionally perform the steps fordetermination.

Hereinafter, the viewpoint video switching process (step S102), therating giving process (step S104), and the information obtaining process(step S106) are described in detail.

First, the viewpoint video switching process (step S102) is described.FIG. 4 is a flowchart of the viewpoint video switching process performedby the terminal device 102.

First, the terminal device 102 determines whether the viewpoint videoswitching mode is the favorite viewpoint video switching mode (1-1), thefavorite subject switching mode (1-2), the recommended viewpoint videoswitching mode (1-3), or the recommended subject switching mode (1-4)(steps S121, S123, and S126).

If the viewpoint video switching mode is the favorite viewpoint videoswitching mode (Yes in step S121), the terminal device 102 requests theserver 103 to deliver a viewpoint video corresponding to a viewpointvideo position specified by the user 106 (step S122).

If the viewpoint video switching mode is the favorite subject switchingmode (Yes in step S123), the terminal device 102 transmits informationabout a subject specified by the user 106 to the server 103 (step S124).Here, information about a subject is subject information, such as thename, the uniform number, or the like of the subject, or pixelinformation about the subject in a viewpoint video specified by the user106 through a touch operation or a mouse operation. Subject information(name, uniform number, or the like) is obtained when the user 106specifies the subject information output to a screen through a touchoperation or a mouse operation, for example. Pixel information about asubject in a viewpoint video includes the positions of pixels of thesubject in the viewpoint video or information about the pixels, forexample.

The terminal device 102 requests the server 103 to deliver a trackingvideo of the subject specified by the user 106 (step S125).

If the viewpoint video switching mode is the recommended viewpoint videoswitching mode (Yes in step S126), the terminal device 102 requests theserver 103 to deliver a recommended viewpoint video generated by theserver 103 (step S127).

If the viewpoint video switching mode is the recommended subjectswitching mode (No in step S126), the terminal device 102 requests theserver 103 to deliver a recommended subject tracking video generated bythe server 103 (step S128).

After step S122, S125, S127 or S128, the terminal device 102 receivesvideo data transmitted from the server 103 and saves the received videodata in the accumulation unit 112 (step S129). Note that the video datareceived from the server 103 may include a video created by the server103 performing processing, editing, or the like in addition to aviewpoint video.

The terminal device 102 need not wait for reception of the entire videodata that is received from the server 103 to be completed and may startreplaying the video at the time of reception of a certain amount ofdata. The terminal device 102 may replay a previously replayed videountil video data is received from the server 103 and replay of the videocan be started.

Next, the terminal device 102 extracts video data for display from theaccumulation unit 112 and decodes the video data (step S130). Then, theterminal device 102 generates a combined video (output video) bycombining the decoded video data with a user interface for display (stepS131).

Next, the terminal device 102 outputs the combined video to the monitor105, such as a display (step S132). As a result, the combined video isdisplayed on the monitor 105.

Hereinafter, the rating giving process (step S104) is described. FIG. 5is a flowchart of the rating giving process performed by the terminaldevice 102.

First, the terminal device 102 determines whether the rating giving modeis the viewpoint video rating giving mode (2-1) or the subject ratinggiving mode (2-2) (step S141).

If the rating giving mode is the viewpoint video rating giving mode (Yesin step S141), the terminal device 102 transmits rating informationabout a viewpoint video that is being viewed by the user 106 and timeinformation about the time when the rating was given to the server 103(step S142). Here, rating information includes a comment or a rating ona viewpoint video. A rating may be information indicating whether theviewpoint video is good or poor or information indicating to what degreethe viewpoint video is good or poor.

If the rating giving mode is the subject rating giving mode (No in stepS141), the terminal device 102 transmits information about a subjectspecified by the user 106 to the server 103 (step S143). Note that thecontent of information about a subject and the method for obtaining theinformation are similar to those in step S124 described above.

Subject information (name, uniform number, or the like) is obtained whenthe user 106 specifies the subject information output to a screenthrough a touch operation or a mouse operation, by voice, or the like,for example. Pixel information about a subject in a viewpoint videoincludes the positions of pixels of the subject in the viewpoint videoor information about the pixels.

Next, the terminal device 102 transmits rating information about thesubject specified by the user 106 and time information about the timewhen the rating was given to the server 103 (step S144).

Hereinafter, the information obtaining process (step S106) is described.FIG. 6 is a flowchart of the information obtaining process performed bythe terminal device 102.

First, the terminal device 102 determines whether the informationobtaining mode is the viewpoint video information obtaining mode (3-1)or the subject information obtaining mode (3-2) (step S161).

If the information obtaining mode is the viewpoint video informationobtaining mode (Yes in step S161), the terminal device 102 transmits anobtaining request for obtaining information about a viewpoint video thatis being viewed by the user 106 to the server 103 (step S162). Here,information about a viewpoint video includes comments or ratings on theviewpoint video, a profile of the videographer of the viewpoint video,information about other content created by the videographer of theviewpoint video, and the like, for example.

If the information obtaining mode is the subject information obtainingmode (No in step S161), the terminal device 102 transmits informationabout a subject specified by the user 106 to the server 103 (step S163).Note that the content of information about a subject and the method forobtaining the information are similar to those in step S124 describedabove.

Next, the terminal device 102 transmits an obtaining request forobtaining information about the subject specified by the user 106 to theserver 103 (step S164). Here, information about a subject includescomments or ratings on the subject, a profile of the subject,information about items related to the subject, and the like, forexample.

After step S162 or S164, the terminal device 102 receives informationabout the viewpoint video or the subject transmitted from the server 103(step S165). Next, the terminal device 102 adds the received informationabout the viewpoint video or the subject to the user interface fordisplay and combines the interface with the video that is being replayedto thereby generate a combined video (step S166). Note that the terminaldevice 102 may display the received information about the viewpointvideo or the subject by superimposing the information on the video thatis being replayed.

Next, the terminal device 102 outputs the generated combined video tothe monitor 105 (step S167). As a result, the combined video isdisplayed on the monitor 105.

Hereinafter, a configuration of the server 103 is described. FIG. 7 is ablock diagram illustrating a configuration of the server 103. The server103 illustrated in FIG. 7 includes a reception unit 121, an accumulationunit 122, an image processing unit 123, a transmission unit 124, and acontrol unit 126.

The reception unit 121 receives from the plurality of cameras 101 aplurality of viewpoint videos obtained by the plurality of cameras 101shooting the same scene over the same time period from differentviewpoints and image capturing camera information. Image capturingcamera information includes position information about the cameras 101,image capture time information, profile information about thevideographers, and the like, for example. Here, position informationabout an image capturing camera is information obtained by using aglobal positioning system (GPS) sensor or the like built in the imagecapturing camera, for example. Note that the position information aboutan image capturing camera may be position information estimated fromradio information about WiFi (registered trademark) communication or thelike or from feature values of the captured video. The server 103 mayestimate the position of the camera 101 from radio information aboutWiFi communication or the like or from feature values of the capturedvideo.

The accumulation unit 122 stores the plurality of viewpoint videos andthe image capturing camera information received by the reception unit121. The accumulation unit 122 stores comments or ratings on viewpointvideos or subjects given by users together with time information aboutthe times when these comments or ratings were given. The accumulationunit 122 stores collection data, such as statistics or the like aboutviewpoint videos or subjects on which a large number of comments havebeen made or which have been highly rated. The accumulation unit 122 maystore information about viewpoint videos or subjects, such as profilesof the videographers of the viewpoint videos or profiles of the subjectsor information about items related to the subjects.

The image processing unit 123 generates video data by processing orediting viewpoint videos in response to the viewpoint video switchingrequest from the terminal device 102 and transmits the generated videodata to the terminal device 102 via the transmission unit 124. The imageprocessing unit 123 identifies a viewpoint video that includes a subjectindicated by pixel information by performing feature value matching ofthe pixel information about the subject in a viewpoint video, the pixelinformation being received from the terminal device 102, with aplurality of viewpoint videos captured by the cameras 101, and transmitsthe result of identification to the control unit 126.

The transmission unit 124 transmits the video data generated by theimage processing unit 123.

The control unit 126 receives data from the terminal device 102 and fromthe Internet (network 104A or 104B). Specifically, the control unit 126receives the viewpoint video switching request, the rating givingrequest, and the information obtaining request from the terminal device102. The control unit 126 creates a search key for a subject, obtains aprofile of the subject, information about related items, and the like bysearching the Internet, and stores the profile or information in theaccumulation unit 122 or transmits the profile or information to theterminal device 102. Further, the control unit 126 transmits/receivesdata to/from a Web store for purchasing related items.

Hereinafter, an operation of the server 103 is described. FIG. 8 is aflowchart of the entire process performed by the server 103.

First, the server 103 receives a plurality of viewpoint videostransmitted from the plurality of cameras 101 and saves the viewpointvideos together with image capturing camera information in theaccumulation unit 122 (step S201).

Next, the server 103 determines whether the viewpoint video switchingrequest has been received from the terminal device 102 (step S202). Ifthe viewpoint video switching request has been received (Yes in stepS202), the server 103 performs a viewpoint video switching process (stepS203).

The server 103 determines whether the rating giving request has beenreceived from the terminal device 102 (step S204). If the rating givingrequest has been received (Yes in step S204), the server 103 performs arating giving process (step S205).

The server 103 further determines whether the information obtainingrequest has been received from the terminal device 102 (step S206). Ifthe information obtaining request has been received (Yes in step S206),the server 103 performs an information obtaining process (step S207).

Note that the order of steps illustrated in FIG. 8 is an example, andthe determination steps in steps S202, S204, and S206 may be performedin any order.

Rating information about a rating given to a viewpoint video or asubject need not be limited to information obtained directly from theuser 106 and may be obtained separately. For example, rating informationmay be separately obtained via a social networking service (SNS) or thelike.

Hereinafter, the viewpoint video switching process (step S203), therating giving process (step S205), and the information obtaining process(step S207) are described in detail.

First, the viewpoint video switching process (step S203) is described.FIG. 9 is a flowchart of the viewpoint video switching process performedby the server 103.

First, the server 103 determines whether the viewpoint video switchingmode is the favorite viewpoint video switching mode (1-1), the favoritesubject switching mode (1-2), the recommended viewpoint video switchingmode (1-3), or the recommended subject switching mode (1-4) (steps S221,S223, and S226).

If the viewpoint video switching mode is the favorite viewpoint videoswitching mode (Yes in step S221), the server 103 selects a viewpointvideo corresponding to a viewpoint video position that has been receivedfrom among viewpoint videos stored in the accumulation unit 122 and setsthe viewpoint video as video data to be transmitted to the terminaldevice 102 (step S222).

If the viewpoint video switching mode is the favorite subject switchingmode (Yes in step S223), the server 103 identifies a subject that hasbeen specified by the user 106 by using information about the subjectreceived from the terminal device 102 (step S224). Specifically, theserver 103 uses subject information (name, uniform number, or the like),or pixel information about the subject in a viewpoint video and aplurality of viewpoint videos from a group of image capturing cameras,and identifies the subject by performing feature value matching or thelike.

Next, the server 103 selects a plurality of viewpoint videos thatinclude the identified subject from among the viewpoint videos stored inthe accumulation unit 122 and switches between the selected viewpointvideos for each period in accordance with the proportion of the area ofthe subject in each video to thereby generate video data to betransmitted to the terminal device 102 (step S225). Note that the server103 may preferentially select a viewpoint video in which the full faceof the subject is always visible or may preferentially select aviewpoint video in which the subject is always visible at the center,for example, instead of selecting a viewpoint video in accordance withthe proportion of the area of the subject in each video. That is, theserver 103 may select a video that provides a good view of the subjectin accordance with the image capture condition.

If the viewpoint video switching mode is the recommended viewpoint videoswitching mode (Yes in step S226), the server 103 automatically switchesbetween viewpoint videos for each period in accordance with statisticaldata about highly rated viewpoint videos to thereby generate video datato be transmitted to the terminal device 102 (step S227).

Here, a highly rated viewpoint video is a viewpoint video that receiveshigh ratings from users. For example, a menu for giving a high rating isprovided for each viewpoint video, and the number of high ratings on theviewpoint video is incremented by one when a user selects the menu. Ahighly rated viewpoint video is a viewpoint video that has the largestnumber of high ratings or that has a number of high ratings larger thana predetermined value. Further, a menu for giving a low rating may beprovided. In this case, the rating of a viewpoint video may bedetermined on the basis of the difference between the number of highratings and the number of low ratings or the like.

A rating using numerical values on a scale from 1 to 5 where 5 indicatesbest and 1 indicates poorest, for example, may be given by a user. Inthis case, the rating of a viewpoint video may be determined on thebasis of the average of the rating values.

Determination described above may take into consideration the number ofratings given (both the number of high ratings and the number of lowratings). Alternatively, a viewpoint video that is replayed (viewed) alarger number of times may be determined to be highly rated, or aviewpoint video that is replayed (viewed) for a longer time may bedetermined to be highly rated.

A menu for a user to make a comment on a viewpoint video may beprovided, and a viewpoint video that receives a larger number ofcomments from users may be determined to be highly rated. Further, thecontent of comments may be analyzed, and a high or low rating, or anumerical value on a scale may be determined on the basis of the contentof the comments.

Alternatively, a viewpoint video that is registered as a favorite by alarger number of users may be determined to be highly rated.

Two or more methods among the methods for determination described abovemay be combined.

The statistical data may be statistical data about the entire scene thatis shot or may be statistical data about a divided scene thatcorresponds to each period. For example, in a case of a sports game, thestatistical data may be statistical data for the entire period of thegame, or may be statistical data for the first half and the second halfor for each predetermined period. Alternatively, the statistical datamay be statistical data for a long period that extends beyond the periodof the scene that is shot. For example, in the case of a sports game,the statistical data may be statistical data about all of a plurality ofgames played in one week or in one month or may be statistical data forone season or for a plurality of seasons.

If the viewpoint video switching mode is the recommended subjectswitching mode (No in step S226), the server 103 identifies a subject byusing statistical data about highly rated subjects (step S228). Notethat a highly rated subject is determined in a manner similar to thatfor a highly rated viewpoint video described above.

Next, the server 103 selects a plurality of viewpoint videos thatinclude the identified subject from among the viewpoint videos stored inthe accumulation unit 122 and switches between the selected viewpointvideos for each period in accordance with the proportion of the area ofthe subject in each video to thereby generate video data to betransmitted to the terminal device 102 (step S229). Note that the server103 may preferentially select a viewpoint video in which the full faceof the subject is always visible or may preferentially select aviewpoint video in which the subject is always visible at the centerinstead of selecting a viewpoint video on the basis of the proportion ofthe area of the subject in each video. That is, the server 103 mayselect a video that provides a good view of the subject in accordancewith the image capture condition.

After step S222, S225, S227, or S229, the server 103 transmits thegenerated video data to the terminal device 102 (step S230).

Note that in a case where a specific event occurs, such as in a casewhere a moving subject becomes non-visible in a viewpoint video in thefavorite viewpoint video switching mode or in the favorite subjectswitching mode or in a case where image capture is discontinued becausethe battery of the camera that is capturing a viewpoint video runs out,the server 103 may detect the event and automatically switch to anotherviewpoint video for viewing that is captured at a position closer tothat of the viewpoint video that the user 106 has been viewing. Theserver 103 may switch the switching mode to the recommended viewpointvideo switching mode or the recommended subject switching mode. On theother hand, the server 103 may switch the switching mode to the favoriteviewpoint video switching mode or the favorite subject switching mode ina case where a specific event occurs in the recommended viewpoint videoswitching mode or the recommended subject switching mode.

Comments and rating values on a subject need not be limited to thosegiven by users, and the server 103 may separately obtain comments andrating values. For example, the server 103 may set in advance an initialvalue on the basis of the number of search results obtained through aWeb search using the name of the subject.

Hereinafter, the rating giving process (step S205) is described. FIG. 10is a flowchart of the rating giving process performed by the server 103.

First, the server 103 determines whether the rating giving mode is theviewpoint video rating giving mode (2-1) or the subject rating givingmode (2-2) (step S241).

If the rating giving mode is the viewpoint video rating giving mode (Yesin steep S241), the server 103 receives rating information that is acomment or a rating on a viewpoint video that is being viewed by theuser 106 and time information about the time when the rating was givenfrom the terminal device 102 and saves the received rating informationabout the viewpoint video in the accumulation unit 122 (step S242).

If the rating giving mode is the subject rating giving mode (No in steepS241), the server 103 identifies a subject by using information aboutthe subject received from the terminal device 102 (step S243). Note thatthe content of information about a subject and the method foridentifying the subject are similar to those in step S224 describedabove.

Next, the server 103 receives rating information about the subjectspecified by the user 106 and time information about the time when therating was given from the terminal device 102 and saves the receivedrating information about the subject in the accumulation unit 122 (stepS244).

After step S242 or S244, the server 103 periodically updates statisticalinformation about highly rated viewpoint videos or subjects by usingrating information about viewpoint videos or subjects and timeinformation about the times when ratings were given, the ratinginformation and time information having been saved in the accumulationunit 122, and saves the updated statistical information in theaccumulation unit 122 (step S245).

Hereinafter, the information obtaining process (step S207) is described.FIG. 11 is a flowchart of the information obtaining process performed bythe server 103.

First, the server 103 determines whether the information obtaining modeis the viewpoint video information obtaining mode (3-1) or the subjectinformation obtaining mode (3-2) (step S261).

If the information obtaining mode is the viewpoint video informationobtaining mode (Yes in step S261), the server 103 reads informationabout a viewpoint video that is being viewed by the user 106 from theaccumulation unit 122 and transmits the read information to the terminaldevice 102 (step S262). Here, information about a viewpoint videoincludes comments or ratings on the viewpoint video, a profile of thevideographer of the viewpoint video, information about other contentcreated by the videographer of the viewpoint video, and the like, forexample.

If the information obtaining mode is the subject information obtainingmode (No in step S261), the server 103 identifies a subject specified bythe user 106 by using information about the subject received from theterminal device 102 (step S263). Note that the content of informationabout a subject and the method for obtaining the information are similarto those in step S224 described above. Alternatively, the server 103 mayselect several subjects by performing feature value matching or the likeand make the user 106 select a target subject from among the severalsubjects.

Next, the server 103 reads information about the subject specified bythe user 106 from the accumulation unit 122 and transmits the readinformation to the terminal device 102 (step S264). Here, informationabout a subject includes comments or ratings on the subject, a profileof the subject, information about items related to the subject, and thelike, for example.

Hereinafter, an example of an operation of the viewpoint video switchingprocess is described. FIG. 12 is a diagram illustrating an example of anoperation in the favorite subject switching mode.

In the example, the server 103 selects a viewpoint video A, a viewpointvideo B, and a viewpoint video C that include a specified subject fromviewpoint videos stored in the accumulation unit 122 and switchesbetween the selected viewpoint videos for each period in accordance withthe proportion of the area of the subject in each video to therebygenerate video data to be transmitted to the terminal device 102. In thefirst period, the proportion of the area of the subject specified by theuser 106 is larger in the viewpoint video A, and therefore, theviewpoint video A is selected in this period. In the next period, theproportion of the area of the subject specified by the user 106 islarger in the viewpoint video C, and therefore, the viewpoint video C isselected in this period. In the last period, the proportion of the areaof the subject specified by the user 106 is larger in the viewpointvideo B, and therefore, the viewpoint video B is selected in thisperiod. In doing so, a video formed by connecting a plurality ofviewpoint videos together in accordance with the area of the subject isgenerated.

Note that the server 103 may perform control so as not to performswitching between viewpoint videos as long as the continuous replay timeof a viewpoint video does not exceed a threshold, that is, until thecontinuous replay time exceeds the threshold. In doing so, it ispossible to suppress frequent switching between viewpoint videos. Forexample, even in a case where the area of the subject becomes larger inthe viewpoint video C than in the viewpoint video A, if the continuousreplay time of the viewpoint video A does not exceed five seconds, forexample, the server 103 keeps replaying the viewpoint video A. Theserver 103 performs switching at a timing of switching to the viewpointvideo C only in a case where the subject is visible in the viewpointvideo C for two seconds or longer, for example.

FIG. 13 is a diagram illustrating an example of an operation in therecommended viewpoint video switching mode. As illustrated in FIG. 13,the server 103 switches between viewpoint videos for each period inaccordance with statistical data about highly rated viewpoint videos tothereby generate video data to be transmitted to the terminal device102. Note that the server 103 may control switching on the basis of thecontinuous replay time similarly to the case of the favorite subjectswitching mode.

FIG. 14 is a diagram illustrating an example of an operation in therecommended subject switching mode. As illustrated in FIG. 14, theserver 103 selects a plurality of viewpoint videos that include a highlyrated subject from among viewpoint videos stored in the accumulationunit 122 and switches between the selected viewpoint videos for eachperiod in accordance with the proportion of the area of the subject ineach video to thereby generate video data to be transmitted to theterminal device 102.

Note that the server 103 may control switching on the basis of thecontinuous replay time similarly to the case of the favorite subjectswitching mode.

Although the process has been described above, for example, where in thecase where switching described above is performed, the server 103generates a single video stream by performing switching and transmitsthe video stream to the terminal device 102, part of the process may beperformed by the terminal device 102. For example, the server 103 maytransmit a plurality of viewpoint videos and information indicatingtimings of switching between the viewpoint videos to the terminal device102, and the terminal device 102 may switch between viewpoint videos forreplay (display) among the plurality of viewpoint videos on the basis ofthe information.

Hereinafter, an example of a display screen of the terminal device 102is described. FIG. 15 is a diagram illustrating an example of an outputscreen of the terminal device 102. FIG. 15 illustrates a state where alive video is displayed.

An output screen 300A illustrated in FIG. 15 includes a viewpoint videodisplay screen 301, a viewpoint selection screen 302, operation buttons303A to 303C, 304A, and 304B, and information display screens 305 and306.

On the viewpoint video display screen 301, a viewpoint video selected bythe user or video data generated by the server 103 in a mode selected bythe user is displayed.

When the user selects a player on the viewpoint video display screen 301by performing a touch or mouse-click operation or the like, a circleoperation menu 307C is displayed. The operation menu 307C includes aplurality of menus, namely, a “high rating” menu, a “low rating” menu, a“player information” menu, and a “favorite” menu, which are arrangedclockwise from the top of the circle. When the user selects the “highrating” or “low rating” menu, a rating (high rating or low rating) isgiven to the selected player (subject rating giving mode).

Note that a rating may be given using any method, such as a method ofgiving a rating as points from one to six points or a method of giving arating as a comment or the like saying “very good”, “good”, or the like.As the method of rating, a method of communicating a comment with apicture or the like may be used as well as with text or by voice.

The values of points may be increased by setting an upper limit on thenumber of points that each user can give. Further, the upper limit ofthe number of points that each user can give may be increased when theuser pays.

When the user selects the “player information” menu, information aboutthe player, such as comments or ratings on the player, information aboutrelated items, or the records of the player, is displayed (subjectinformation obtaining mode).

When the user selects the “favorite” menu, a viewpoint video at the timeof selection or around the time of selection may be registered as afavorite play of the selected player (subject rating giving mode).

The server 103 or the terminal device 102 may access a website or thelike related to the information about related items, the records of theplayer, or other information about the player, and the terminal device102 may display the content of the website or the like. The server 103or the terminal device 102 may access a website or the like from whichrelated items can be purchased, and the terminal device 102 may displaythe content of the website. As a result, the viewer can easily purchasean item while watching a game. Accordingly, the provider of this servicecan connect the viewer with the item selling company via the service andcan expect an effect of earning a profit from the item selling company,for example.

When the user selects the “favorite” menu, the selected player isregistered as a favorite player of the user. A player registered in thefavorite subject switching mode is used as a favorite subject.

Although the example where only one viewpoint video display screen 301is present is illustrated here, a plurality of viewpoint video displayscreens may be displayed simultaneously. For example, three viewpointvideo display screens may be included in an output screen, a viewpointvideo in which the entire court is visible may be displayed on a firstviewpoint video display screen, a viewpoint video specified by the usermay be displayed on a second viewpoint video display screen (favoriteviewpoint video switching mode), and a viewpoint video of a subjectspecified by the user may be displayed on a third viewpoint videodisplay screen (favorite subject switching mode). On the viewpoint videodisplay screen 301, a video created by combining a plurality ofviewpoint videos may be displayed. In this case, when the user changesthe angle right and left and up and down while keeping touching thescreen, for example, multi-angle viewing is enabled.

On the viewpoint video display screen 301, a highly rated player may behighlighted and displayed. For example, the outline of a highly ratedplayer may be illuminated and displayed.

The viewpoint selection screen 302 shows a diagram illustrating thepositions of the plurality of cameras 101 (viewpoint positions ofviewpoint videos). In the example illustrated in FIG. 15, the positionsof the plurality of cameras 101 are represented by circles on the topview of the game venue, for example.

The position represented by the hatched circle indicates the viewpointposition of the viewpoint video that is currently being displayed on theviewpoint video display screen 301. When the user selects the circlecorresponding to each viewpoint position by performing a touch ormouse-click operation or the like, the viewpoint video that is displayedis switched to the viewpoint video corresponding to the viewpointposition (favorite viewpoint video switching mode).

Note that the viewpoint position of a highly rated viewpoint video maybe highlighted and displayed on the viewpoint selection screen 302. Theserver 103 may analyze audio data about cheers and the like measured ateach viewpoint position, and a viewpoint position at which loudestcheers are heard at the time may be highlighted and displayed on theviewpoint selection screen 302. Further, switching to and display of aviewpoint video in which loudest cheers are heard may be performed foreach period.

When the user selects a viewpoint position on the viewpoint selectionscreen 302 by keeping touching or mouse-clicking the viewpoint position,a circle operation menu 307A is displayed. Operations related to theoperation menu 307A are similar to those related to the above-describedoperation menu 307C except that the target is a viewpoint video insteadof a player (subject).

Specifically, the operation menu 307A includes a plurality of menus,namely, a “high rating” menu, a “low rating” menu, a “viewpoint videoinformation” menu, and a “favorite” menu, which are arranged clockwisefrom the top of the circle. When the user selects the “high rating” or“low rating” menu, a rating (high rating or low rating) is given to theselected viewpoint video (viewpoint video rating giving mode). Note thata rating may be given using any method, such as a method of giving arating as points from one to six points or a method of giving a ratingas a comment or the like saying “very good”, “good”, or the like.

When the user selects the “viewpoint video information” menu, commentsor ratings on the viewpoint video, a profile of the videographer of theviewpoint video, information about other content created by thevideographer of the viewpoint video, or the like are displayed(viewpoint video information obtaining mode).

When the user selects the “favorite” menu, a viewpoint video at the timeof selection or around the time of selection may be registered as afavorite time period of the selected viewpoint video (viewpoint videorating giving mode). The registered favorite time period is used in acase of recording viewing. For example, in the case of recordingviewing, another viewpoint video corresponding to the favorite timeperiod is displayed.

When the user selects the “favorite” menu, the selected viewpoint videois registered as a favorite video. The registered favorite video is usedas a viewpoint video in the favorite viewpoint video switching mode.

The operation buttons 303A to 303C are buttons for selecting thefavorite subject switching mode, the recommended subject switching mode,and the recommended viewpoint video switching mode respectively.

The operation buttons 304A and 304B are buttons for switching between alive video and a recorded video. For example, when the user selects theoperation button 304A, a live video is displayed on the viewpoint videodisplay screen 301. When the user selects the operation button 304B, arecorded video is displayed on the viewpoint video display screen 301.

On the information display screen 305, information about the currentgame or the like is displayed. For example, the numbers in the circlesillustrated in FIG. 15 represent the uniform numbers of the players. Forexample, the uniform numbers on the upper row represent the uniformnumbers of the players who are currently playing in the game, and theuniform numbers on the lower row represent the uniform numbers of backupplayers.

When the user selects the uniform number of a player by performing atouch or mouse-click operation or the like, a circle operation menu 307Bis displayed. Operations related to the operation menu 307B are similarto those related to the above-described operation menu 307C.

On the information display screen 306, a summary or the like of ratingson subjects (players) given by users is displayed. Similarly to theinformation display screen 305, the numbers in the circles represent theuniform numbers of the players. To the right of each uniform number,ratings on the player having the uniform number given by a plurality ofusers are indicated. Specifically, the numbers of high ratings areindicated on the upper row, and the numbers of low ratings are indicatedon the lower row. In the example illustrated in FIG. 15, the highratings and low ratings are each given on the basis of six levels (1 to6), and the number of given ratings is indicated for each level. Theseratings are summarized for each predetermined period, and the summaryvalues are displayed.

On the left edge of the information display screen 306, the longitudinaldirection is assumed to represent time, and the times at whichcorresponding scenes are registered as favorite scenes are indicated bystars.

When the user clicks on the uniform number of a player, comments andratings given to the player are displayed in detail.

The summary values displayed on the information display screen 306 maybe updated at a certain interval. Alternatively, the summary values maybe updated each time a rating is given by a user. Points given to asubject may be fed back to the subject by reflecting the points in theannual salary of the subject, for example.

FIG. 16 is a diagram illustrating an example of an output screen of theterminal device 102. FIG. 16 illustrates a state where a recorded videois displayed. For example, an output screen 300B illustrated in FIG. 16is displayed when the user selects the operation button 304B on theoutput screen 300A illustrated in FIG. 15.

The output screen 300B illustrated in FIG. 16 includes an informationdisplay screen 308 instead of the information display screen 305 in theoutput screen 300A illustrated in FIG. 15.

The lateral direction of the information display screen 308 representstime, and the degree of excitement (volume of cheers or the like, forexample) is represented by a wave. The stars on the lower row representthe times at which corresponding scenes are registered as favoritescenes. The circles on the middle row represent the times of recommendedscenes. The numbers in the circles on the lower row represent theuniform numbers of the players of one team who have gained points, andthe numerical values immediately above the respective uniform numbersrepresent the scores. Similarly, the numbers in the circles on the upperrow represent the uniform numbers of the players of the other team whohave gained points, and the numerical values immediately below therespective uniform numbers represent the scores.

In a case where a viewpoint video that is distributed as a recommendedvideo is viewed by a larger number of people, points may be given to thevideographer of the viewpoint video.

When a recorded video is displayed, view points may be displayed inaccordance with a viewpoint video or subject registered by users as afavorite, a highly rated viewpoint video or subject, or audio data ofcheers or the like.

In a case where a subject is selected, a series of plays correspondingto ratings given to the subject may be displayed as a ranking list orthe like, and the series of plays may be replayed. For example, in acase where the user selects a player having the uniform number 5, aseries of plays in which a larger number of points (for example, sixpoints) have been given to the player having the uniform number 5 isdisplayed, and one play selected from among the plays is replayed.

In a case of displaying a comment related to each rating, a list ofviewpoint videos captured by a user who has made the comment may bedisplayed simultaneously. A videographer may be registered as a favoriteor the like. In this case, a mode may be provided in which onlyviewpoint videos captured by the videographer registered as a favoritecan be viewed.

FIG. 17 is a diagram illustrating another example of an output screenaccording to this embodiment. An output screen 300C illustrated in FIG.17 includes a main screen 311, a first sub-screen 312A, a secondsub-screen 312B, and a viewpoint selection screen 313.

On the main screen 311, a viewpoint video specified by the user isdisplayed (favorite viewpoint video switching mode). On the firstsub-screen 312A, a recommended viewpoint video generated in theabove-described recommended viewpoint video switching mode is displayed.On the second sub-screen 312B, a recommended subject video generated inthe above-described recommended subject switching mode is displayed.

When the user clicks on the first sub-screen 312A in this state, therecommended viewpoint video is displayed on the main screen 311, and asecondary recommended viewpoint video is displayed on the firstsub-screen 312A.

Similarly, when the user clicks on the second sub-screen 312B, therecommended subject video is displayed on the main screen 311, and asecondary recommended subject video is displayed on the secondsub-screen 312B.

In a case where the user specifies a subject in the viewpoint videodisplayed on the main screen 311 by performing a click operation or thelike, the mode transitions to the favorite subject switching mode, and atracking video of the specified subject is displayed. When the userclicks on the main screen 311 while the favorite subject video is beingdisplayed, the display is switched to the viewpoint video that has beenpreviously viewed.

Here, although the example is illustrated where a recommended viewpointvideo and a recommended subject video are displayed on the firstsub-screen 312A and the second sub-screen 312B respectively, a firstrecommended viewpoint video may be displayed on the first sub-screen312A, and a second recommended viewpoint video may be displayed on thesecond sub-screen 312B. Similarly, a first recommended subject video maybe displayed on the first sub-screen 312A, and a second recommendedsubject video may be displayed on the second sub-screen 312B.

The display on the first sub-screen 312A and the second sub-screen 312Bmay be turned off, and the information display screen 305, 306, or 308illustrated in FIG. 15 or FIG. 16 may be displayed.

When the user double-clicks on the main screen 311, for example, onlythe main screen 311 may be displayed on the display in full-screen mode.

Note that the viewpoint selection screen 313 is similar to the viewpointselection screen 302 illustrated in FIG. 15 and FIG. 16.

Hereinafter, a method for delivering videos (tracking videos) edited bythe server 103 to the terminal device 102 is described. First, theserver 103 generates a first tracking video of a subject A and a secondtracking video of a subject B. Here, although the example of generatingtwo tracking videos is described for simplifying the description, threeor more tracking videos may be generated. A video transmitted from theserver 103 is not limited to a tracking video, and may be one of theviewpoint videos or may be a favorite viewpoint video, a recommendedviewpoint video, or the like.

The server 103 transmits the plurality of generated videos to theterminal device 102 through multicasting (broadcasting). Specifically,the server 103 delivers the first tracking video on the first channeland the second tracking video on the second channel. The terminal device102 switches between the videos for display by switching between thechannels in accordance with a user request.

By using this method, a needed band can be made constant. As a result,it is possible to easily handle a case where there are a large number ofusers (a large number of terminal devices 102).

Although there is an issue of difficulty in handling a case of a largenumber of users because of the varying needed band, one video desired bya user may be transmitted from the server 103 to a certain terminaldevice 102 or may be transmitted and received between terminal devices102. In a case of a smaller number of users, a needed band can be madenarrower, which is an advantage.

Hereinafter, a process to be performed in a case where a communicationerror occurs is described.

Ideally, the server 103 generates recommended viewpoint videos orrecommended subject videos in accordance with the results of ratingsgiven by users. However, it is not possible to generate or transmitthese videos in the following cases.

First, the following cases may arise between the server 103 and theterminal device 102.

(1) One of the cases is a case where a transmission video (recommendedviewpoint video, recommended subject video, or the like) has a largeamount of data relative to a reception band that can be used by theterminal device 102. In this case, any of the following methods can beemployed.

(1-1) The server 103 lowers the resolution or the frame rate of thetransmission video, re-encodes the transmission video, and transmits theresulting video to the terminal device 102. In doing so, a transmissionvideo having a reduced amount of data is used.

Note that the server 103 may generate a plurality of video signalshaving respective bit rates by converting the resolution or the framerate of a viewpoint video transmitted from the camera 101 and store theplurality of video signals in advance.

In a case where the bandwidth usable by the terminal device 102 changesduring viewing, the server 103 may switch between transmission videos inaccordance with the usable bandwidth.

The server 103 may determine the number of videos to be transmitted inaccordance with a communication band usable by the terminal device 102.Specifically, the server 103 increases the number of transmission videosas the communication band becomes wider.

(1-2) The server 103 uses another video that is similar to the targetvideo (for example, another video captured at a position closer to thatof the target video or another video including subjects similar to thosein the target video) and that has a smaller amount of data as asubstitution.

(2) The other of the cases is a case where a video that is not allowedto be distributed or a region that is not allowed to be viewed as aresult of distribution is included in a transmission video or aviewpoint video used to generate a transmission video. In this case, anyof the following methods can be employed.

(2-1) The server 103 uses another video that is similar to the targetvideo (for example, another video captured at a position closer to thatof the target video or another video including subjects similar to thosein the target video) and that has no problem as a substitution.

(2-2) The server 103 uses another highly rated video as a substitution.For example, in a case where a video having the second highest ratingdoes not have a similar problem, the server 103 uses the video as asubstitution.

Further, the following cases may arise between the camera 101(videographer) and the server 103.

Specifically, (3-1) a case where a video from the camera 101 does notarrive at the server 103 due to packet loss or the like, (3-2) a casewhere the band of the camera 101 is unstable and data transfer to theserver 103 is delayed, and (3-3) a case where the videographer suddenlydiscontinues capturing a video (because the battery runs out or athermal issue arises, for example) may occur. In these cases, any of thefollowing methods can be employed.

(4-1) The server 103 uses a highly rated viewpoint video or anothervideo that is similar to the subject video (for example, another videocaptured at a position closer to that of the subject video or anothervideo including subjects similar to those in the subject video) as asubstitution.

(4-2) The server 103 uses another highly rated video as a substitution.For example, the server 103 uses a video having the second highestrating as a substitution.

Hereinafter, a method for transmitting a video other than videos highlyrated by users is described.

Here, an unpopular viewpoint video is a video not viewed by a largenumber of people, that is, a video gaining little attention.Specifically, an unpopular viewpoint video is a video that is replayed asmall number of times, a video having a low rating, a video on which asmall number of ratings or comments have been given, or the like.However, such a viewpoint video may be valuable to some people, andtherefore, the server 103 may connect viewpoint videos gaining littleattention together and deliver the connected viewpoint videos. Forexample, the system described above can be used for surveillance purpose(applied to a surveillance system including a plurality of surveillancecameras). Here, a case is assumed where videos from the plurality ofsurveillance cameras are monitored by a plurality of persons. In thiscase, although a viewpoint video watched by the plurality of persons issufficiently monitored, a viewpoint video not watched by a sufficientnumber of persons is likely to cause a blind spot. Therefore, the server103 connects viewpoint videos gaining little attention (viewpoint videoswatched by a small number of persons) together and delivers theconnected viewpoint videos. The connected videos are displayed on aseparate screen beside the viewpoint video that the guards are watching.Accordingly, it is possible to reduce blind spots that the guards failto monitor.

In a case where there is a viewpoint video that does not gain attentionfor a certain time period, a notification or warning about the viewpointvideo may be sent to the guards, or information indicating the viewpointvideo not gaining attention may be communicated to the guards. Here, aviewpoint video that does not gain attention for a certain time periodis a viewpoint video that is not replayed for a certain time period, forexample. In a case where a plurality of viewpoint videos aresimultaneously displayed, a viewpoint video that does not gain attentionfor a certain time period may be a viewpoint video that is not selectedas a video to be displayed on the main screen for a certain time periodor a viewpoint video that is not given a rating for a certain timeperiod. A viewpoint video that does not gain attention for a certaintime period may be a video that is replayed a number of times equal toor smaller than a predetermined threshold, that is selected a number oftimes equal to or smaller than a predetermined threshold, or that isgiven a number of ratings equal to or smaller than a predeterminedthreshold.

For example, on the viewpoint selection screen 302 illustrated in FIG.15 or the like, the viewpoint position of a viewpoint video that doesnot gain attention for a certain time period may be highlighted anddisplayed. For example, the color of the viewpoint position may bechanged to a color different from those of other viewpoint positions. Asa result, it is possible to urge the guards to monitor the viewpointvideo that does not gain attention.

As describe above, the server 103 according to this embodiment performsthe process illustrated in FIG. 18.

The server 103 delivers to the terminal device 102 a first video signalbased on a plurality of videos obtained by shooting the same scene froma plurality of different viewpoints over a plurality of successiveperiods.

First, the server 103 selects a first selected video from among aplurality of videos for each of the plurality of periods on the basis ofrating values given to the plurality of videos by a plurality of users(step S301). Specifically, the server 103 selects a video having a highrating value as the first selected video.

More specifically, as described in the above description of therecommended viewpoint video switching mode, a rating value is a ratingvalue of each of the plurality of videos, and the server 103 selects,for each of the plurality of periods, a video having the highest ratingvalue or having a rating value higher than a predetermined value in theperiod from among the plurality of videos as the first selected video.

Alternatively, as described in the above description of the recommendedsubject switching mode, a rating value is a rating value of each of aplurality of subjects included in the plurality of videos, and theserver 103 selects, for each of the plurality of periods, a video inwhich a recommended subject having the highest rating value or having arating value higher than a predetermined value is visible from among theplurality of videos as the first selected video. For example, the server103 selects a video in which a recommended subject having a largest areais visible from among the plurality of videos as the first selectedvideo.

Alternatively, as described in the above description of the method fortransmitting a video other than videos highly rated by users, a ratingvalue is a rating value of each of the plurality of videos, and theserver 103 may select, for each of the plurality of periods, a videohaving a rating value lower than a predetermined value in the periodfrom among the plurality of videos as the first selected video.

Next, the server 103 transmits to the terminal device 102 the firstvideo signal for successively replaying a plurality of first selectedvideos respectively selected for the plurality of periods (step S302).Specifically, the server 103 generates a video signal by connecting aplurality of selected videos respectively selected for the plurality ofperiods together and transmits the video signal to the terminal device102. Alternatively, the server 103 transmits, as a video signal, theplurality of videos and information for identifying selected videosrespectively selected for the plurality of periods to the terminaldevice 102. In this case, the terminal device 102 generates the videosignal by connecting the plurality of selected videos respectivelyselected for the plurality of periods together by using the information.

As described in the above description of the process to be performed inthe case where a communication error occurs, if the amount of data ofthe first video signal is larger than an upper limit based on thereception band of the terminal device 102, the server 103 furtherselects a second selected video from among the plurality of videos foreach of the plurality of periods on the basis of the rating values ofthe plurality of videos and transmits to the terminal device 102 asecond video signal for successively replaying a plurality of secondselected videos respectively selected for the plurality of periods.Here, at least one of the second selected videos respectively selectedfor the plurality of periods is different from the first selected videosrespectively selected for the plurality of periods, and the amount ofdata of the second selection signal is smaller than the upper limit.

The plurality of videos are transmitted from the plurality of cameras101 to the server 103 in real time, and the server 103 selects, in acase where transmission of a video corresponding to a first selectedvideo from the camera 101 is discontinued, a second selected videodifferent from the first selected video and transmits a second videosignal for replaying the second selected video to the terminal device102.

The terminal device 102 according to this embodiment performs theprocess illustrated in FIG. 19.

The terminal device 102 receives from the server 103 a video signalbased on a plurality of videos obtained by shooting the same scene overa plurality of successive periods from a plurality of differentviewpoints.

First, the terminal device 102 receives from the server 103 a videosignal for successively replaying a plurality of selected videosselected from among the plurality of videos for the plurality of periodson the basis of the rating values of the plurality of videos given by aplurality of users (step S311). Specifically, the selected videos arevideos having high rating values.

More specifically, as described in the above description of therecommended viewpoint video switching mode, a rating value is a ratingvalue of each of the plurality of videos, and a selected video is avideo that is selected from among the plurality of videos for each ofthe plurality of periods and that has the highest rating value or arating value higher than a predetermined value in the period.

Alternatively, as described in the above description of the recommendedsubject switching mode, a rating value is a rating value of each of aplurality of subjects included in the plurality of videos, and aselected video is a video which is selected from among the plurality ofvideos for each of the plurality of periods and in which a subjecthaving the highest rating value or having a rating value higher than apredetermined value is visible.

Alternatively, as described in the above description of the method fortransmitting a video other than videos highly rated by users, a ratingvalue is a rating value of each of the plurality of videos, and aselected video is a video that is selected from among the plurality ofvideos for each of the plurality of periods and that has a rating valuelower than a predetermined value in the period.

Next, the terminal device 102 successively replays the plurality ofselected videos in accordance with the received video signal (stepS312). Specifically, the terminal device 102 receives the video signalgenerated by the server 103 connecting the plurality of selected videosrespectively selected for the plurality of periods together andreproduces the video signal. Alternatively, the terminal device 102receives from the server 103 the plurality of videos and information foridentifying each selected video selected for each of the plurality ofperiods, generates a video signal by connecting the plurality ofselected videos respectively selected for the plurality of periodstogether by using the information, and reproduces the video signal.

The terminal device 102 may display any of the plurality of videos andmay highlight and display a subject having the highest rating value or arating value higher than a predetermined value among a plurality ofsubjects included in the displayed video.

As illustrated in FIG. 15, the terminal device 102 may display any ofthe plurality of videos and, in a case where any subject is selected bythe user from among a plurality of subjects included in the displayedvideo, may display the operation menu 307C for rating the selectedsubject.

The terminal device 102 may display any of the plurality of videos and,in a case where any subject is selected by the user from among aplurality of subjects included in the displayed video, may display alist of videos for the plurality of periods among the plurality ofvideos, the videos in the list including the selected subject and havingrating values higher than a predetermined value.

As illustrated in FIG. 17, the terminal device 102 may display any ofthe plurality of videos and, in a case where any subject is selected bythe user from among a plurality of subjects included in the displayedvideo, may display a tracking video that is a video obtained byconnecting a plurality of videos together for tracking the selectedsubject.

As illustrated in FIG. 17, the terminal device 102 may display theoutput screen 300C that includes a first screen (main screen 311) onwhich any of the plurality of videos is displayed and a second screen(first sub-screen 312A or second sub-screen 312B) which is smaller thanthe first screen and on which a plurality of selected videos that arereplayed are displayed, and may display the plurality of selected videoson the first screen in a case where the second screen is selected by theuser.

Note that part of the process performed by the server 103 may beperformed by the terminal device 102, and part of the process performedby the terminal device 102 may be performed by the server 103. That is,the video delivery system 100 according to this embodiment reproduces(for display) a video signal based on a plurality of videos obtained byshooting the same scene over a plurality of successive periods from aplurality of different viewpoints. The video delivery system 100 selectsa selected video from among the plurality of videos for each of theplurality of periods on the basis of the rating values of the pluralityof videos given by a plurality of users. The video delivery system 100generates a video signal by connecting the plurality of selected videosrespectively selected for the plurality of periods together andreproduces the video signal (for display).

Although the video delivery method, the video reception method, and thevideo delivery system according to this embodiment have been described,the present disclosure is not limited to this embodiment.

Each of the processing units of each apparatus included in the videodelivery system according to the above-described embodiment is typicallyimplemented as a large scale integration (LSI) circuit, which is anintegrated circuit (IC). The processing units may each be implemented asone chip, or some or all of the processing units may be implemented asone chip.

Integration into a circuit may be implemented by using a dedicatedcircuit or a general-purpose processor instead of an LSI circuit.Alternatively, a field programmable gate array (FPGA) that isprogrammable after manufacturing the LSI circuit or a reconfigurableprocessor for which connections or configurations of circuit cellsinside the LSI circuit are reconfigurable may be used.

In the above-described embodiment, each constituent element may beconstituted by a dedicated hardware unit or may be implemented byexecuting a software program suitable to the constituent element. Eachconstituent element may be implemented by a program execution unit, suchas a central processing unit (CPU) or a processor, reading and executinga software program recorded to a recording medium, such as a hard diskor a semiconductor memory.

In other words, each apparatus included in the video delivery systemincludes processing circuitry and storage electrically connected to theprocessing circuitry (accessible from the processing circuitry). Theprocessing circuitry includes at least one of the dedicated hardwareunit and the program execution unit. In a case where the processingcircuitry includes the program execution unit, the storage storestherein a software program executed by the program execution unit. Theprocessing circuitry performs the video delivery method or the videoreception method according to the above-described embodiment by usingthe storage.

Alternatively, the present disclosure may relate to the above-describedsoftware program or a non-transitory computer-readable recording mediumto which the program is recorded. The program may be distributed througha transmission medium, such as the Internet, as a matter of course.

Any numbers mentioned above are provided as examples in order tospecifically describe the present disclosure and are not intended torestrict the present disclosure.

The above-described order of performing the steps included in the videodelivery method or the video reception method is an example forspecifically describing the present disclosure, and the steps may beperformed in other orders than the above-described order. Alternatively,some of the steps may be performed simultaneously with (in parallel to)other steps.

Although the video delivery method, the video reception method, thevideo delivery system, the server, and the terminal device according toone or more aspects of the present disclosure have been described withreference to the embodiment, the present disclosure is not limited tothe embodiment. Various modifications that are conceivable by a personskilled in the art and that are made on the embodiment, or a formconfigured by combining constituent elements in different embodimentsmay also fall within the scope of one or more aspects of the presentdisclosure without departing from the spirit of the present disclosure.

Second Embodiment

Examples of other applications of the configurations of the imageprocessing methods and apparatuses described in the above-describedembodiment and a system that employs the applications will be described.The system is applicable to video systems that become more intelligentand that target wider space, such as (1) a surveillance system installedon security cameras in stores or plants, onboard cameras mounted inpolice cars, and the like, (2) a traffic information system usingprivately owned cameras, onboard cameras, cameras placed along roads,and the like, (3) an environment survey system or delivery system usingremotely operable or automatically controllable devices, such as drones,and (4) a content transmission/reception system fortransmitting/receiving videos and the like using cameras placed inamusement facilities, stadiums, or the like, moving cameras mounted ondrones and the like, privately owned cameras, or the like, for example.

FIG. 20 is a diagram illustrating a video information processing systemex100 according to this embodiment. In this embodiment, an example ofpreventing a blind spot from being created and an example of prohibitingimage capture of a specific area will be described.

The video information processing system ex100 illustrated in FIG. 20includes a video information processing device ex101, a plurality ofcameras ex102, and a video reception device ex103. Note that the videoreception device ex103 need not be included in the video informationprocessing system ex100.

The video information processing device ex101 includes a saving unitex111 and an analysis unit ex112. Each of N cameras ex102 has a functionof capturing a video and a function of transmitting data of the capturedvideo to the video information processing device ex101. The camera ex102may have a function of displaying a video that is being captured. Notethat the camera ex102 may transmit a video signal of the captured videoto the video information processing device ex101 after performingencoding using a coding scheme, such as High Efficiency Video Coding(HEVC) or H.264, or may transmit video data that is not encoded to thevideo information processing device ex101.

Here, the cameras ex102 are each a fixed camera, such as a surveillancecamera, a moving camera mounted on a radio-controlled pilotlessaircraft, a vehicle, or the like, or a user camera owned by a user.

A moving camera receives an instruction signal transmitted from thevideo information processing device ex101 and changes its position orimage capture direction in accordance with the received instructionsignal.

The times set on the plurality of cameras ex102 are calibrated by usingtime information from a server or a reference camera before startingimage capture. The spatial positions of the plurality of cameras ex102are calibrated on the basis of how an object in space, which is an imagecapture target, is viewed or on the basis of the relative positions fromthe reference camera.

In the saving unit ex111 included in the video information processingdevice ex101, video data transmitted from the N cameras ex102 is saved.

The analysis unit ex112 detects a blind spot from the video data savedin the saving unit ex111 and transmits an instruction signal indicatingan instruction to a moving camera for eliminating the blind spot to themoving camera. The moving camera moves in accordance with theinstruction signal and continues image capture.

The analysis unit ex112 detects a blind spot by using a Structure fromMotion (SfM) scheme. An SfM scheme is a technique for restoring thethree-dimensional form of a subject from a plurality of videos capturedfrom different positions and is generally known as a form restorationtechnique for simultaneously estimating the form of a subject and thepositions of the cameras. For example, the analysis unit ex112 restoresthe three-dimensional form of a facility or a stadium from video datasaved in the saving unit ex111 by using an SfM scheme and detects anarea for which restoration fails as a blind spot.

In a case where the positions and image capture directions of thecameras ex102 are fixed and information about the positions and imagecapture directions are known, the analysis unit ex112 may perform an SfMprocess by using the known information. In a case where the position andimage capture direction of a moving camera can be obtained from a GPS,an angle sensor, and the like mounted on the moving camera, the movingcamera may transmit information about the position and image capturedirection to the analysis unit ex112, and the analysis unit ex112 mayperform an SfM process by using the transmitted information about theposition and image capture direction.

The method for detecting a blind spot is not limited to the method ofusing an SfM scheme described above. For example, the analysis unitex112 may grasp the spatial distance to a subject, which is an imagecapture target, by using information from a depth sensor, such as alaser rangefinder. The analysis unit ex112 may detect information aboutthe position, image capture direction, zoom factor, and the like of acamera by determining whether a marker or a specific object set inadvance in space is included in the image and, if the marker or specificobject is included, on the basis of the size or the like of the markeror specific object. In doing so, the analysis unit ex112 detects a blindspot by using any method with which an image capture area of each cameracan be detected. The analysis unit ex112 may obtain information aboutpositional relationships or the like among a plurality of image capturetargets from video data or from a proximity sensor or the like andidentify an area in which a blind spot is highly likely to be created onthe basis of the obtained positional relationships.

Here, examples of a blind spot include not only a portion for which avideo is not obtained in an area for which an image is to be capturedbut also a portion having a poor image quality compared to otherportions, a portion not satisfying a predetermined image quality, andthe like. Such a portion that is to be detected may be set appropriatelyin accordance with the configuration or purpose of the system. Forexample, an image quality required for a specific subject in imagecapture space may be increased. On the other hand, an image qualityrequired for a specific area in image capture space may be decreased.Setting may be performed so as not to determine a portion for which avideo has not been captured to be a blind spot.

Note that the image quality described above includes various types ofinformation about a video, such as information about the area (thenumber of pixels, for example) of a subject, which is an image capturetarget, in the video or information about whether a subject, which is animage capture target, is brought into focus. Determination as to whethera portion becomes a blind spot may be performed on the basis of any ofthese pieces of information or a combination thereof.

Although detection of an area that is actually a blind spot has beendescribed above, an area that needs to be detected in order to prevent ablind spot from being created is not limited to an area that is actuallya blind spot. For example, in a case where there are a plurality ofimage capture targets and at least some of the image capture targets aremoving, an image capture target may come between another image capturetarget and a camera, which may create a new blind spot. In this case,the analysis unit ex112 may detect movements of the plurality of imagecapture targets from obtained video data or the like and may estimate anarea that is likely to become a new blind spot on the basis of thedetected movements of the plurality of image capture targets andposition information about the cameras ex102, for example. Here, thevideo information processing device ex101 may transmit an instructionsignal to a moving camera for instructing the moving camera to capturean image of an area that is likely to become a blind spot to therebyprevent a blind spot from being created.

In a case where there are a plurality of moving cameras, the videoinformation processing device ex101 needs to select a moving camera towhich an instruction signal for capturing an image of a blind spot or anarea that is likely to become a blind spot is transmitted. In a casewhere there are a plurality of moving cameras and there are a pluralityof blind spots or a plurality of areas that are likely to become blindspots, the video information processing device ex101 needs to determine,for each of the plurality of moving cameras, for which of the blindspots or for which of the areas that are likely to become blind spots animage is to be captured by the moving camera. For example, the videoinformation processing device ex101 selects a moving camera nearest to ablind spot or an area that is likely to become a blind spot on the basisof the position of the blind spot or the area that is likely to become ablind spot and the positions of areas that the respective moving camerasare capturing images. The video information processing device ex101 maydetermine, for each of the moving cameras, whether a new blind spot iscreated in a case where video data that is currently being obtained bythe moving camera becomes not obtainable, and may select a moving camerafor which a blind spot is not created even if video data that iscurrently being obtained becomes not obtainable.

With the configuration described above, the video information processingdevice ex101 can detect a blind spot and transmits an instruction signalto a moving camera for eliminating a blind spot to thereby prevent ablind spot from being created.

First Modification

Although the example has been described in the above description wherean instruction signal for instructing a moving camera to move istransmitted, the instruction signal may be a signal for instructing theuser of a user camera to move. For example, the user camera displays aninstruction image for instructing the user to change the direction ofthe camera. The user camera may display an instruction image indicatinga movement path on a map as an instruction for the user to move. Theuser camera may display instructions on the details of image capture,such as the image capture direction, angle, angle of view, imagequality, and a movement of the image capture area for increasing thequality of images to be obtained, and the video information processingdevice ex101 may automatically control feature values of the camerasex102 related to the details of image capture if the feature values arecontrollable by the video information processing device ex101.

Here, the user camera is a smartphone, a tablet terminal, a wearableterminal, or a head-mounted display (HMD) carried by a spectator in astadium or a guard in a facility.

A display terminal on which an instruction image is displayed need notbe a user camera that captures a video. For example, a user camera maytransmit an instruction signal or an instruction image to a displayterminal that is associated in advance with the user camera, and thedisplay terminal may display the instruction image. Information about adisplay terminal corresponding to a user camera may be registered inadvance in the video information processing device ex101. In this case,the video information processing device ex101 directly transmits aninstruction signal to the display terminal corresponding to the usercamera to thereby display an instruction image on the display terminal.

Second Modification

The analysis unit ex112 may generate a free-viewpoint video(three-dimensional reconfiguration data) by using an SfM scheme, forexample, and restoring the three-dimensional form of the facility or thestadium from video data saved in the saving unit ex111. Thefree-viewpoint video is saved in the saving unit ex111. The videoinformation processing device ex101 reads from the saving unit ex111video data corresponding to field-of-view information (and/or viewpointinformation) transmitted from the video reception device ex103 andtransmits the video data to the video reception device ex103. Note thatthe video reception device ex103 may be one of the plurality of camerasex102.

Third Modification

The video information processing device ex101 may detect an imagecapture prohibition area. In this case, the analysis unit ex112 analyzesa captured image and, if a moving camera is capturing an image of animage capture prohibition area, transmits an image capture prohibitionsignal to the moving camera. The moving camera stops capturing an imageas long as the moving camera receives an image capture prohibitionsignal.

The analysis unit ex112 matches three-dimensional virtual space restoredby using an SfM scheme with a captured video to thereby determinewhether a moving camera set in advance in space is capturing an image ofan image capture prohibition area. Alternatively, the analysis unitex112 determines whether a moving camera is capturing an image of animage capture prohibition area while using a marker or a characteristicobject placed in space as a trigger. An image capture prohibition areais a toilet or the like in a facility or a stadium, for example.

In a case where a user camera is capturing an image of an image captureprohibition area, the user camera may display a message on a display orthe like connected to the user camera through wired or wirelessconnections or output a beep or sound from a speaker or an earphone tothereby notify the user that the current place is an image captureprohibition place.

For example, as the above-described message, a message is displayedindicating that image capture in the direction in which the camera iscurrently facing is prohibited. Alternatively, an image captureprohibition area and the current image capture area are indicated on amap that is displayed. Image capture is restarted automatically when animage capture prohibition signal is no longer output, for example.Alternatively, image capture may be restarted in a case where an imagecapture prohibition signal is no longer output and the user performs anoperation for restarting image capture. In a case where image capture isstopped and restarted a plurality of times in a short period,calibration may be performed again. A notification for making a usercheck the current position or urging the user to move may be sent.

In a case of special operations, such as police operations, a passcode,fingerprint authentication, or the like may be used to turn off theabove-described function in order to allow recording. Even in such acase, if a video of an image capture prohibition area is externallydisplayed or saved, image processing for blurring or the like may beautomatically performed.

With the configuration described above, the video information processingdevice ex101 determines whether image capture is prohibited and notifiesa user that image capture is to be stopped to thereby set the area as animage capture prohibition area.

Fourth Modification

Videos from a plurality of viewpoints need to be collected in order tostructure three-dimensional virtual space from videos, and therefore,the video information processing system ex100 provides an incentive to auser who has transferred a captured video. For example, the videoinformation processing device ex101 delivers a video at no cost or at adiscount to a user who has transferred a captured video or gives theuser points having a monetary value which can be used in an online oroffline shop or game or points having a nonmonetary value such as asocial status in virtual space, such as a game. The video informationprocessing device ex101 specifically gives a high point to a user whohas transferred a captured video corresponding to a valuable field ofview (and/or viewpoint) for which a larger number of requests are made,for example.

Fifth Modification

The video information processing device ex101 may transmit additionalinformation to a user camera on the basis of the result of an analysisperformed by the analysis unit ex112. In this case, the user camerasuperimposes the additional information on a captured video and displaysthe result on a screen. Additional information is information about aplayer, such as the name, height, or the like of the player, in a casewhere a video is captured for a game played in a stadium, for example,and the name, a photograph of the face, or the like of the player isdisplayed in association with the player in the video. Note that thevideo information processing device ex101 may extract additionalinformation by a search via the Internet on the basis of a partial areaor the entire area of the video data. The camera ex102 may receive suchadditional information through short-range wireless communication basedon Bluetooth (registered trademark) or the like or through visible lightcommunication from lighting in a stadium or the like and may map thereceived additional information on video data. The camera ex102 mayperform such mapping on the basis of a certain rule indicated by a tablethat is retained in a storage unit connected to the camera ex102 throughwireless or wired connections and that indicates a correspondencebetween information obtained using a visible light communicationtechnique and additional information or may perform such mapping byusing the most probable combination obtained as a result of a search viathe Internet.

In a surveillance system, information about an individual to be aware ofis superimposed, for example, for a user camera carried by a guard in afacility to thereby make the surveillance system more sensitive.

Sixth Modification

The analysis unit ex112 matches a free-viewpoint video with a videocaptured by a user camera to thereby determine for which area in thefacility or the stadium an image is being captured by the user camera.The method for determining an image capture area is not limited to this,and the various methods for determining an image capture area describedin the above-described embodiments or other methods for determining animage capture area may be used.

The video information processing device ex101 transmits a past video toa user camera on the basis of the result of an analysis performed by theanalysis unit ex112. The user camera superimposes the past video on acaptured video or replaces the captured video with the past video, anddisplays the result on a screen.

For example, during a half-time break, highlights in the first half aredisplayed as a past video. Accordingly, a user can enjoy the highlightsin the first half as a video in the direction in which the user islooking. Note that the past video is not limited to a video ofhighlights in the first half and may be a video of highlights in a pastgame played in the stadium. A timing at which the video informationprocessing device ex101 delivers a past video is not limited to a timeduring a half-time break, and the video information processing deviceex101 may deliver a past video after the end of the game or during thegame, for example. In a case of delivery of a past video during thegame, the video information processing device ex101 may deliver a videoof a scene that may be missed by a user and that is assumed to beimportant on the basis of the result of an analysis performed by theanalysis unit ex112. The video information processing device ex101 maydeliver a past video only in a case where a user makes a request or maydeliver a message related to delivery permission before delivering apast video.

Seventh Modification

The video information processing device ex101 may transmit advertisementinformation to a user camera on the basis of the result of an analysisperformed by the analysis unit ex112. The user camera superimposes theadvertisement information on a captured video and displays the result ona screen.

Advertisement information may be delivered immediately before deliveringa past video during a half-time break or after the end of the gamedescribed in the sixth modification. As a result, the distributor canearn advertising revenues from the advertiser and can provide a videodelivery service to a user at a low price or at no cost. The videoinformation processing device ex101 may deliver a message related toadvertisement delivery permission immediately before deliveringadvertisement information, may provide a service at no cost only in acase where a user views an advertisement, or may provide a service at alower price compared to a case where a user does not view anadvertisement.

When a user clicks on “order now” or the like in accordance with anadvertisement, a staff member or an automatic delivery system of thesite which grasps the position of the user by using the system or anyposition information delivers an ordered drink to the user's seat.Payment may be made by handing over money to the staff member or byusing credit card information set in advance in an application or thelike of a mobile terminal. An advertisement may include a link to ane-commerce website and may enable online shopping that provides usualdoor-to-door delivery or the like.

Eighth Modification

The video reception device ex103 may be one of the cameras ex102 (usercameras). In this case, the analysis unit ex112 matches a free-viewpointvideo with a video captured by a user camera to thereby determine forwhich area in the facility or the stadium the user camera is capturingan image. The method for determining an image capture area is notlimited to this.

For example, when a user performs a swipe operation in a direction of anarrow displayed on a screen, the user camera generates viewpointinformation indicating that the viewpoint is to be moved in thedirection. The video information processing device ex101 reads from thesaving unit ex111 video data obtained by capturing an image of an areathat is moved, in accordance with the viewpoint information, from theimage capture area of the user camera which is determined by theanalysis unit ex112 and starts transmitting the video data to the usercamera. The user camera displays the video delivered from the videoinformation processing device ex101 instead of the captured video.

In doing so, a user in a facility or a stadium can view a video from aviewpoint that the user likes by performing a simple operation, such asa swipe on a screen. For example, a spectator that watches a game on thethird-base side of a ballpark can view a video from a viewpoint on thefirst-base side. In a surveillance system, a guard in a facility canview a video from a viewpoint that the guard wants to check, a videothat is to be observed as instructed by a center in an interruptingmanner, or the like by adaptively change the viewpoint by a simpleoperation, such as a swipe on a screen. Accordingly, the surveillancesystem can be made more sensitive.

Delivery of a video to a user in a facility or a stadium is effectivealso in a case where an obstacle is present between the user camera andan image capture target and there is an area that cannot be viewed, forexample. In this case, the user camera may switch a video of a partialarea of the image capture area of the user camera, the partial areaincluding the obstacle, from the captured video to a video deliveredfrom the video information processing device ex101 and display thedelivered video or may switch the display from the captured video to thedelivered video and display the delivered video in full-screen mode.Alternatively, the user camera may display a video in which a viewtarget is seen through an obstacle by combining the captured video withthe delivered video. With this configuration, even in a case where animage capture target cannot be viewed from the position of a user due toan obstacle, the user can view a video delivered from the videoinformation processing device ex101, and therefore, an effect of theobstacle can be reduced.

In a case where a delivered video is displayed as a video of an areathat cannot be viewed due to an obstacle, display switching controldifferent from the above-described display switching control that isbased on an input process performed by a user, such as a swipe on ascreen, may be performed. For example, in a case where it is determinedthat an obstacle is included in an image capture area on the basis ofinformation about a movement of a user camera and the image capturedirection of the user camera and position information about an obstaclewhich is obtained in advance, display switching from the captured videoto a delivered video may be automatically performed. In a case where itis determined that an obstacle that is not an image capture target isvisible as a result of an analysis of captured video data, displayswitching from the captured video to a delivered video may beautomatically performed. In a case where the area (for example, thenumber of pixels) of an obstacle included in a captured video exceeds apredetermined threshold or in a case where the ratio of the area of anobstacle to the area of an image capture target exceeds a predeterminedratio, display switching from the captured video to a delivered videomay be automatically performed.

Display switching from a captured video to a delivered video and displayswitching from a delivered video to a capture video may be performed inaccordance with an input process performed by a user.

Ninth Modification

The speed for transferring video data obtained by each camera ex102 tothe video information processing device ex101 may be indicated on thebasis of the degree of importance of the video data.

In this case, the analysis unit ex112 determines the degree ofimportance of video data saved in the saving unit ex111 or the degree ofimportance of the camera ex102 that has obtained the video data. Here,the degree of importance is determined on the basis of the number ofpersons or the number of moving objects included in the video,information about the image quality or the like of the video data, or acombination thereof, for example.

The degree of importance of video data may be determined on the basis ofthe position of the camera ex102 where the video data has been obtainedor the image capture area of the video data. For example, in a casewhere a plurality of other cameras ex102 that are capturing images arepresent near a target camera ex102, the degree of importance of videodata obtained by the target camera ex102 is decreased. Even in a casewhere the target camera ex102 is located away from the other camerasex102, if there are a plurality of other cameras ex102 that arecapturing images of the same area as that for which the target cameraex102 is capturing an image, the degree of importance of video dataobtained by the target camera ex102 is decreased. The degree ofimportance of video data may be determined on the basis of whether alarger number of requests are made in the video delivery service. Themethod for determining the degree of importance is not limited to thosedescribed above or a combination thereof and may be any method as longas the method is suitable to the configuration or purpose of thesurveillance system or video delivery system.

The degree of importance need not be determined on the basis of obtainedvideo data. For example, the degree of importance of the camera ex102that transmits video data to a terminal other than the video informationprocessing device ex101 may be made higher. To the contrary, the degreeof importance of the camera ex102 that transmits video data to aterminal other than the video information processing device ex101 may bemade lower. As a result, in a case where a plurality of services thatneed to transmit video data share a communication band, thecommunication band can be controlled in a more flexible manner inaccordance with the purpose or characteristics of each service.Accordingly, it is possible to prevent the quality of each service frombeing compromised due to unavailability of necessary video data.

The analysis unit ex112 may determine the degree of importance of videodata by using a free-viewpoint video and a video captured by the cameraex102.

The video information processing device ex101 transmits a communicationspeed indication signal to the camera ex102 on the basis of the resultof determination of the degree of importance performed by the analysisunit ex112. The video information processing device ex101 indicates ahigh communication speed for the camera ex102 that is capturing a videoof a high degree of importance, for example. The video informationprocessing device ex101 may transmit a signal not only for controllingthe speed but also for indicating, for important information, a schemefor transmitting the information a plurality of times in order to reducedisadvantages caused by loss. As a result, communication throughout afacility or a stadium can be efficiently performed. Note thatcommunication between the cameras ex102 and the video informationprocessing device ex101 may be wired communication or wirelesscommunication. The video information processing device ex101 may onlycontrol one of wired communication and wireless communication.

The camera ex102 transmits captured video data to the video informationprocessing device ex101 at a communication speed indicated by thecommunication speed indication signal. In a case where the camera ex102fails to retransmit captured video data a predetermined number of times,the camera ex102 may stop retransmitting the captured video data andstart transferring the subsequent captured video data. As a result,communication throughout a facility or a stadium can be efficientlyperformed, and processing in the analysis unit ex112 can be performed ata higher speed.

In a case where a band assigned to the camera ex102 is not sufficientfor transferring captured video data at an indicated communicationspeed, the camera ex102 may convert the captured video data into videodata corresponding to a bit rate which can be transmitted at theindicated communication speed and transmit the converted video data, ormay stop transferring the captured video data.

In a case where video data is used to prevent a blind spot from beingcreated as described above, only a partial area of an image capture areaincluded in captured video data may be needed in order to eliminate ablind spot. In this case, the camera ex102 may generate extraction videodata by extracting only the area needed to prevent a blind spot frombeing created and transmit the generated extraction video data to thevideo information processing device ex101. With this configuration, itis possible to suppress creation of a blind spot with a narrowercommunication band.

In a case where additional information is superimposed and displayed ora video is delivered, for example, the camera ex102 needs to transmitposition information and image capture direction information about thecamera ex102 to the video information processing device ex101. In thiscase, the camera ex102 that is assigned a band insufficient fortransferring video data may only transmit position information and imagecapture direction information which are detected by the camera ex102. Ina case where the video information processing device ex101 estimatesposition information and image capture direction information about thecamera ex102, the camera ex102 may convert captured video data intovideo data having a resolution required for estimating positioninformation and image capture direction information and transmit theconverted video data to the video information processing device ex101.With this configuration, a service of superimposing and displayingadditional information or delivering a video can be provided to thecamera ex102 that is assigned a narrow communication band. Further, thevideo information processing device ex101 can obtain information aboutimage capture areas from a larger number of cameras ex102, which is alsoeffective in a case of using information about image capture areas inorder to detect an area that gains attention, for example.

Switching between video data transfer processes in accordance with anassigned communication band described above may be performed by thecamera ex102 on the basis of the assigned communication band, or thevideo information processing device ex101 may determine an operation ofeach camera ex102 and transmit a control signal indicating thedetermined operation to each camera ex102. As a result, processing canbe appropriately divided and performed in accordance with the processingload for determining switching between operations, the processingcapacity of the camera ex102, a needed communication band, and the like.

Tenth Modification

The analysis unit ex112 may determine the degree of importance of videodata on the basis of field-of-view information (and/or viewpointinformation) transmitted from the video reception device ex103. Forexample, the analysis unit ex112 makes the degree of importance ofcaptured video data that includes a larger area indicated by thefield-of-view information (and/or viewpoint information) higher. Theanalysis unit ex112 may determine the degree of importance of video databy taking into consideration the number of persons or the number ofmoving objects included in the video. The method for determining thedegree of importance is not limited to this.

Note that the communication control method described in this embodimentneed not be used in a system that reconstructs a three-dimensional formfrom a plurality of pieces of video data. For example, in an environmentwhere a plurality of cameras ex102 are present, in a case where piecesof video data are transmitted selectively or by changing thetransmission speed through wired communication and/or wirelesscommunication, the communication control method described in thisembodiment is effective.

Eleventh Modification

In the video delivery system, the video information processing deviceex101 may transmit a panorama video showing the entire scene that isshot to the video reception device ex103.

Specifically, when the video information processing device ex101 hasreceived a delivery request transmitted from the video reception deviceex103, the video information processing device ex101 reads a panoramavideo of the entire facility or stadium from the saving unit ex111 andtransmits the panorama video to the video reception device ex103. Thepanorama video may be updated at a longer interval (may have a low framerate) and may have a low image quality. A viewer touches a portion thatthe viewer wants to view closely in the panorama video displayed on thescreen of the video reception device ex103. Then, the video receptiondevice ex103 transmits field-of-view information (and/or viewpointinformation) corresponding to the touched portion to the videoinformation processing device ex101.

The video information processing device ex101 reads video datacorresponding to the field-of-view information (and/or viewpointinformation) from the saving unit ex111 and transmits the video data tothe video reception device ex103.

The analysis unit ex112 generates a free-viewpoint video bypreferentially performing three-dimensional form restoration(three-dimensional reconstruction) on the area indicated by thefield-of-view information (and/or viewpoint information). The analysisunit ex112 restores the three-dimensional form of the entire facility orstadium with a precision that is sufficient to show a panorama view.Accordingly, the video information processing device ex101 canefficiently perform three-dimensional form restoration. As a result, ahigher frame rate and a higher image quality can be attained for afree-viewpoint video of an area that a viewer wants to view.

Twelfth Modification

The video information processing device ex101 may save in advancethree-dimensional form restoration data of a facility or a stadium whichhas been generated beforehand from design drawings or the like as apreparatory video. A preparatory video is not limited to this and may bevirtual space data obtained by mapping, for each object, irregularitiesin space obtained from a depth sensor with a picture derived from a pastvideo, a video at the time of calibration, or video data.

For example, in a case where soccer is being played in a stadium, theanalysis unit ex112 may perform three-dimensional form restoration onlyon a player and the ball and combine obtained restoration data with thepreparatory video to thereby generate a free-viewpoint video.Alternatively, the analysis unit ex112 may perform three-dimensionalform restoration preferentially on a player and the ball. Accordingly,the video information processing device ex101 can efficiently performthree-dimensional form restoration. As a result, a higher frame rate anda higher image quality can be attained for a free-viewpoint videorelated to the player that the viewer pays attention and the ball. In asurveillance system, the analysis unit ex112 may performthree-dimensional form restoration only on a person and a moving objector preferentially on a person and a moving object.

Thirteenth Modification

The time set on each apparatus may be calibrated on the basis of thereference time of a server or the like when image capture starts. Theanalysis unit ex112 may perform three-dimensional form restoration byusing, among a plurality of pieces of video data obtained by theplurality of cameras ex102, a plurality of pieces of video data obtainedat times within a time range set in advance, in accordance with theprecision of time setting. For detecting the times, the times when theobtained pieces of video data were saved in the saving unit ex111 arereferred to, for example. Note that the method for detecting times isnot limited to this. As a result, the video information processingdevice ex101 can efficiently perform three-dimensional form restoration.Accordingly, a higher frame rate and a higher image quality can beattained for a free-viewpoint video.

The analysis unit ex112 may perform three-dimensional form restorationby only using high-image-quality data or preferentially usinghigh-image-quality data among a plurality of pieces of video data savedin the saving unit ex111.

Fourteenth Modification

The analysis unit ex112 may perform three-dimensional form restorationby using camera attribute information. In this case, the camera ex102transmits captured video data and camera attribute information to thevideo information processing device ex101. Camera attribute informationincludes the image capture position, image capture angle, image capturetime, zoom factor, and the like.

As a result, the video information processing device ex101 canefficiently perform three-dimensional form restoration. Accordingly, ahigher frame rate and a higher image quality can be attained for afree-viewpoint video.

Specifically, the camera ex102 defines three-dimensional coordinates ina facility or a stadium and transmits to the video informationprocessing device ex101 information indicating when an image of whichlocation indicated by coordinates was captured by the camera ex102 fromwhat angle with what zoom factor as camera attribute informationtogether with the video. At the time of activation of the camera ex102,the clock of the communication network of the facility or the stadium issynchronized with the clock of the camera ex102, and time information isgenerated.

When the camera ex102 is oriented to a specific point in the facility orstadium at the time of activation of the camera ex102 or at any time,position information and angle information about the camera ex102 areobtained. FIG. 21 is a diagram illustrating an example of a notificationdisplayed on the screen of the camera ex102 at the time of activation ofthe camera ex102. When the user positions the “+” sign displayed at thecenter of the screen over the “+” sign at the center of the soccer ballon a billboard placed on the north side of the stadium in accordancewith the notification and touches the display of the camera ex102, thecamera ex102 obtains vector information about a vector from the cameraex102 to the billboard and identifies the references of the cameraposition and angle. Thereafter, the camera coordinates and angle areidentified at the time from motion information about the camera ex102.As a matter of course, the display is not limited to this, and displaymay be used in which the coordinates, the angle, the moving speed of theimage capture area, or the like are indicated by arrows or the likeduring image capture.

The coordinates of the camera ex102 may be identified by using a radiowave of GPS, WiFi (registered trademark), 3G, Long Term Evolution (LTE),or 5G (wireless LAN) communication or may be identified by usingshort-range wireless communication, such as a beacon (Bluetooth(registered trademark) or ultrasound). Alternatively, informationindicating at which base station in the facility or stadium capturedvideo data has arrived may be used.

Fifteenth Modification

The system may be provided as an application that operates on a mobileterminal, such as a smartphone.

Accounts of various SNSs or the like may be used in logging in to thesystem. An account exclusively used for the application or a guestaccount having a limited function may be used. By using accounts in thisway, favorite videos, favorite accounts, or the like can be rated. Bypreferentially assigning a band to video data similar to video data thatis being obtained or viewed, video data corresponding to a viewpointsimilar to that of video data that is being obtained or viewed, or thelike, the resolution of such video data can be increased. As a result,three-dimensional form restoration from such a viewpoint can beperformed with a higher precision.

When a user selects a favorite image or video and follows thecounterpart in the application, the user can view the selected imagemore preferentially than other users or can form a connection with thecounterpart through text chats or the like on the condition of agreementby the counterpart. As a result, a new community can be created.

As described above, when users are connected with one another in acommunity, image capture itself or sharing of captured images areactively performed, and three-dimensional form restoration with a higherprecision can be promoted.

A user can edit an image or a video captured by another person or createa new collage image or video by using another person's image and theuser's image in accordance with settings of connections in thecommunity. As a result, new video works can be shared, that is, a newimage or video can be shared only by people in the community, forexample. The video works can be used for augmented reality (AR) games orthe like by inserting a computer graphics (CG) character when editing animage or a video.

In the system, three-dimensional model data can be output at any time,and therefore, a 3D printer or the like owned by the facility can outputa 3D object on the basis of three-dimensional model data correspondingto a characteristic scene, such as a goal scene. As a result, objectsbased on the scene in the game can be sold as souvenirs, such as keychains, or can be distributed to participating users. As a matter ofcourse, an image from the best viewpoint can be printed as a usualphotograph.

Sixteenth Modification

By using the above-described system, the general situation of the entireregion can be managed by a center connected to the system on the basisof videos from onboard cameras mounted in police cars, wearable camerasof police officers, and the like, for example.

In a case of usual patrol, still images are transmitted and receivedevery few minutes, for example. The center identifies a region wherecrimes are highly likely to occur on the basis of a crime map based onthe result of an analysis using past crime data or the like, or retainsregional data related to the probability of crime occurrence thusidentified. For the identified region where crimes are highly likely tooccur, images may be transmitted and received more frequently or imagesmay be changed to moving images. Upon the occurrence of a crime, amoving image or three-dimensional reconfiguration data generated byusing an SfM scheme may be used. When the center or each terminalsimultaneously corrects an image or virtual space by using informationfrom other sensors, such as depth sensors or thermo-sensors, policeofficers can grasp the situation more accurately.

The center can feed back information about an object of interest to aplurality of terminals by using three-dimensional reconfiguration data.As a result, the individuals having the terminals can track the object.

Currently, image capture from the air by using apparatuses capable offlying, such as quadcopters, drones, and the like, are performed forsurveying buildings or environments or capturing vivid images of sportsgames or the like, or for other purposes. Although image capture withsuch autonomous moving apparatuses tends to have a problem of blurredimages, it is possible to perform three-dimensional form restorationwith an SfM scheme while correcting the blurs on the basis of theposition and tilt. As a result, the image quality and the precision ofspace restoration can be increased.

In some countries, installation of onboard cameras that capture imagesof scenes outside the vehicles are required. Regarding such onboardcameras, the weather, the conditions of roads, the degree of trafficcongestion, and the like in the direction of the destination can begrasped more accurately by using three-dimensional data that is modeledby using a plurality of images.

Third Embodiment

The processes described in the above-described embodiments can be easilyimplemented in a standalone computer system by recording a program forimplementing the configuration of the image processing methods describedin the embodiments to a storage medium. The storage medium may be anytype of medium capable of storing the program, such as a magnetic disk,an optical disc, a magneto-optical disk, an IC card, or a semiconductormemory.

Hereinafter, examples of applications of the image processing methodsdescribed in the above-described embodiments and systems that employ theapplications will be further described. The systems include apparatusesthat employ the image processing methods. Other configurations in thesystems can be changed as appropriate in accordance with thecircumstances.

FIG. 22 is a diagram illustrating an overall configuration of a contentproviding system ex200 that implements a content delivery service. Anarea in which a communication service is provided is divided into cellsof a desired size. Base stations ex206, ex207, ex208, ex209, and ex210,which are fixed radio stations, are installed in the respective cells.

In the content providing system ex200, various devices, such as acomputer ex211, a personal digital assistant (PDA) ex212, a cameraex213, a smartphone ex214, and a game machine ex215, are connected tothe Internet ex201 via an Internet service provider ex202, acommunication network ex204, and the base stations ex206 to ex210.

Note that the configuration of the content providing system ex200 is notlimited to the configuration illustrated in FIG. 22, and any combinationof the elements may be connected. Further, the individual devices may bedirectly connected to the communication network ex204, such as atelephone network, a cable television network, or an opticalcommunication network, instead of via the base stations ex206 to ex210,which are fixed radio stations. Alternatively, the individual devicesmay be directly interconnected via short-range wireless communication orthe like.

The camera ex213 is a device capable of capturing moving images, such asa digital video camera. A camera ex216 is a device capable of capturingstill images and moving images, such as a digital camera. The smartphoneex214 may be any type of smartphone that complies with Global System forMobile Communications (GSM) (registered trademark), Code DivisionMultiple Access (CDMA), Wideband-Code Division Multiple Access (W-CDMA),LTE, High Speed Packet Access (HSPA), or other communication schemesusing a high-frequency band, a Personal Handyphone System (PHS)terminal, or the like.

In the content providing system ex200, the camera ex213 or the like isconnected to a streaming server ex203 via the base station ex209 and thecommunication network ex204 to thereby enable live streaming. Duringlive streaming, a coding process is performed on content (for example, avideo of a music event or the like) obtained by the user capturing avideo using the camera ex213, and the resulting content is transmittedto the streaming server ex203. The streaming server ex203 in turndelivers the transmitted content data as a stream to a client that hasmade a request. Examples of the client include the computer ex211, thePDA ex212, the camera ex213, the smartphone ex214, and the game machineex215 capable of decoding the data that has undergone the codingprocess. Each device that has received the delivered data performs adecoding process on the received data and reproduces the data.

Note that the coding process may be performed on the obtained data bythe camera ex213, by the streaming server ex203 that performs a datatransmission process, or the coding process may be divided and thedivided portions may be performed by the camera ex213 and the streamingserver ex203 respectively. Similarly, the decoding process may beperformed on the delivered data by the client, by the streaming serverex203, or the decoding process may be divided and the divided portionsmay be performed by the client and the streaming server ex203respectively. In addition to still and/or moving image data obtained bythe camera ex213, still and/or moving image data obtained by the cameraex216 may be transmitted to the streaming server ex203 via the computerex211. In this case, the coding process may be performed by any of thecamera ex216, the computer ex211, and the streaming server ex203, or thecoding process may be divided and the divided portions may be performedby the camera ex216, the computer ex211, and the streaming server ex203respectively. Regarding display of the decoded image, a plurality ofdevices connected in the system may display the same image inconjunction with one another, or a device having a large display unitmay display an image of the entire view, and the smartphone ex214 or thelike may enlarge and display a partial area of the image.

The coding and decoding processes are typically performed by an LSIex500 included in the computer ex211 or in each device. The LSI ex500may be formed of a single chip or a plurality of chips. Alternatively,software for moving image coding/decoding may be recorded to a recordingmedium (such as a compact disc read-only memory (CD-ROM), a flexibledisk, or a hard disk) that is readable by the computer ex211 or thelike, and the coding and decoding processes may be performed using thesoftware. Further, in a case where the smartphone ex214 is equipped witha camera, moving image data obtained with the camera may be transmitted.This moving image data is data that has been coded by the LSI ex500included in the smartphone ex214.

The streaming server ex203 may be constituted by a plurality of serversor a plurality of computers that process, record, and deliver data in adistributed manner.

As described above, the content providing system ex200 allows the clientto receive and reproduce coded data. Accordingly, the content providingsystem ex200 allows the client to receive, decode, and reproduceinformation transmitted by a user in real time, and therefore, allows auser having no special right or equipment to realize personalbroadcasting.

The above-described embodiments may be applied not only to the exampleof the content providing system ex200 but also to a digital broadcastingsystem ex300, as illustrated in FIG. 23. Specifically, a broadcastingstation ex301 transmits multiplexed data obtained by multiplexing musicdata and the like with video data to a broadcasting satellite ex302using a radio wave. The video data is data coded by using a moving imagecoding method. The broadcasting satellite ex302 that receives the dataemits a broadcasting radio wave, and a home antenna ex304 capable ofreceiving satellite broadcasting receives the radio wave. A device, suchas a television (receiver) ex400 or a set top box (STB) ex317, decodesand reproduces the received multiplexed data.

Further, a video decoding apparatus or a video coding apparatus can beimplemented in a reader/recorder ex318 that reads and decodesmultiplexed data recorded to a recording medium ex315, such as a digitalversatile disc (DVD) or a Blu-ray disc (BD), or to a memory ex316, suchas an SD card, or that codes a video signal, further multiplexes a musicsignal with the video signal depending on the circumstances, and writesthe resulting signal to the recording medium ex315 or the memory ex316.In this case, the reproduced video signal is displayed on a monitorex319, and the video signal can be reproduced by another apparatus orsystem using the recording medium ex315 or the memory ex316 having themultiplexed data recorded thereto. Alternatively, a video decodingapparatus may be implemented in the STB ex317 connected to a cable ex303for cable television or the home antenna ex304 for satellite/terrestrialbroadcasting, and the video may be displayed on the monitor ex319 of thetelevision ex400. At this time, a video decoding apparatus may beincorporated into the television ex400 instead of the STB ex317.

FIG. 24 is a diagram illustrating the smartphone ex214. FIG. 25 is adiagram illustrating an example of a configuration of the smartphoneex214. The smartphone ex214 includes an antenna ex450 that transmits andreceives a radio wave to and from the base station ex210, a camera unitex465 capable of capturing videos and still images, and a display unitex458, such as a liquid crystal display (LCD), that displays the videoscaptured by the camera unit ex465 and data obtained by decoding videosor the like received via the antenna ex450. The smartphone ex214 furtherincludes an operation unit ex466 such as a touch panel, an audio outputunit ex457 such as a speaker for outputting sound, an audio input unitex456 such as a microphone for inputting sound, a memory unit ex467 thatstores coded data or decoded data of captured videos, captured stillimages, recorded sound, received videos, received still images, orreceived emails, and a slot unit ex464, which serves an interface with asubscriber identity module (SIM) ex468 for identifying the memory ex316illustrated in FIG. 23 or a user and performing authentication foraccess to a network and various types of data.

The smartphone ex214 includes a main control unit ex460 that controlsthe display unit ex458 and the operation unit ex466 in an integratedmanner. The smartphone ex214 also includes a power supply circuit unitex461, an operation input control unit ex462, a video signal processingunit ex455, a camera interface unit ex463, an LCD control unit ex459, amodulation/demodulation unit ex452, a multiplexing/demultiplexing unitex453, an audio signal processing unit ex454, the slot unit ex464, andthe memory unit ex467 which are connected to the main control unit ex460via a bus ex470.

When the on-hook/power key is turned on through a user operation, thepower supply circuit unit ex461 supplies electric power to theindividual units from a battery pack to thereby activate the smartphoneex214 into an operable state.

In the smartphone ex214, in a voice call mode, the audio signalprocessing unit ex454 converts an audio signal obtained by the audioinput unit ex456 into a digital audio signal, themodulation/demodulation unit ex452 performs spread spectrum processingon the digital audio signal, and a transmission/reception unit ex451performs digital-to-analog conversion processing and frequencyconversion processing on the signal and thereafter transmits theresulting signal via the antenna ex450 in accordance with controlperformed by the main control unit ex460 which includes a CPU, aread-only memory (ROM), and a random access memory (RAM). Further, inthe smartphone ex214, in the voice call mode, the transmission/receptionunit ex451 amplifies reception data received via the antenna ex450 andperforms frequency conversion processing and analog-to-digitalconversion processing, the modulation/demodulation unit ex452 performsde-spread spectrum processing on the resulting signal, and the audiosignal processing unit ex454 converts the resulting signal into ananalog audio signal. Thereafter, the analog audio signal is output fromthe audio output unit ex457.

In a case where an email is transmitted in a data communication mode,text data of the email input through operation of the operation unitex466 or the like is sent to the main control unit ex460 via theoperation input control unit ex462. The main control unit ex460 performscontrol so as to make the modulation/demodulation unit ex452 performspread spectrum processing on the text data and to make thetransmission/reception unit ex451 perform digital-to-analog conversionprocessing and frequency conversion processing on the text data andthereafter transmit the resulting text data to the base station ex210via the antenna ex450. In a case of receiving an email, substantiallythe opposite processing is performed on the received data, and theresulting data is output to the display unit ex458.

In a case where a video, a still image, or a combination of a video andsound is transmitted in the data communication mode, the video signalprocessing unit ex455 compresses and codes a video signal supplied fromthe camera unit ex465 by using a video coding method, and sends thecoded video data to the multiplexing/demultiplexing unit ex453. Theaudio signal processing unit ex454 codes an audio signal obtained by theaudio input unit ex456 while the video, still image, or the like isbeing captured by the camera unit ex465, and sends the coded audio datato the multiplexing/demultiplexing unit ex453.

The multiplexing/demultiplexing unit ex453 multiplexes the coded videodata supplied from the video signal processing unit ex455 with the codedaudio data supplied from the audio signal processing unit ex454 inaccordance with a predetermined scheme. The modulation/demodulation unit(modulation/demodulation circuit unit) ex452 performs spread spectrumprocessing on the resulting multiplexed data. The transmission/receptionunit ex451 performs digital-to-analog conversion processing andfrequency conversion processing on the multiplexed data and thereaftertransmits the resulting data via the antenna ex450.

In a case of receiving data of a moving image file linked to a websiteor the like or in a case of receiving an email attached with a videoand/or sound in the data communication mode, themultiplexing/demultiplexing unit ex453 demultiplexes multiplexed datainto a bitstream of video data and a bitstream of audio data in order todecode the multiplexed data received via the antenna ex450. Themultiplexing/demultiplexing unit ex453 supplies the coded video data tothe video signal processing unit ex455 and the coded audio data to theaudio signal processing unit ex454 via the synchronization bus ex470.The video signal processing unit ex455 performs decoding using a videodecoding method to thereby decode the video signal. Then, a video or astill image included in the moving image file linked to the website isdisplayed on the display unit ex458 via the LCD control unit ex459, forexample. The audio signal processing unit ex454 decodes the audiosignal, and the resulting sound is output from the audio output unitex457.

The smartphone ex214 or other terminals may be implemented in threeforms, that is, a transmission/reception terminal including both anencoder and a decoder, a transmission terminal only including anencoder, and a reception terminal only including a decoder, similarly tothe television ex400. Although the case has been described wheremultiplexed data in which music data and the like is multiplexed withvideo data is received and transmitted in the digital broadcastingsystem ex300, the multiplexed data may be data in which text datarelated to the video other than audio data is multiplexed, or video dataalone may be used instead of the multiplexed data.

The present disclosure is not limited to the above-describedembodiments, and various modifications and alternations can be madewithout departing from the scope of the present disclosure.

The present disclosure is applicable to a video delivery system thatdelivers videos captured by a plurality of cameras.

What is claimed is:
 1. A method comprising: obtaining a plurality of videos by taking a subject from a plurality of different viewpoints over a plurality of successive periods; selecting a plurality of first selected videos respectively selected for each of the plurality of periods, each of the plurality of first selected videos being selected from among the plurality of videos for each of the plurality of periods in accordance with rating values of the plurality of videos, the rating values being given by a plurality of users; transmitting from a server to a terminal device, a first video signal for successively replaying the plurality of first selected videos; and when the first video signal has an amount of data larger than an upper limit based on a reception band of the terminal device: selecting a plurality of second selected videos respectively selected for the plurality of periods, each of the plurality of second selected videos being selected from among the plurality of videos for each of the plurality of periods in accordance with the rating values of the plurality of videos; and transmitting to the terminal device, instead of the first video signal, a second video signal for successively replaying the plurality of second selected videos, wherein at least one of the plurality of second selected videos respectively selected for the plurality of periods is different from the plurality of first selected videos respectively selected for the plurality of periods, the second video signal has an amount of data smaller than the upper limit, and each of a plurality of viewpoints of the selected second videos is closer to the subject than each of a plurality of viewpoints of the selected first videos.
 2. The method according to claim 1, wherein each of the rating values is a rating value of each of the plurality of videos, and in the selecting, selecting each of the plurality of first selected videos having a highest rating value or having a rating value higher than a predetermined value for a respective period.
 3. The method according to claim 1, wherein each of the rating values is a rating value of each of a plurality of subjects included in the plurality of videos, and in the selecting, selecting each of the plurality of first selected videos in which a recommended subject having a highest rating value or having a rating value higher than a predetermined value is visible.
 4. The method according to claim 3, wherein in the selecting, selecting each of the plurality of first selected videos in which the recommended subject occupies a largest area within a display.
 5. The method according to claim 1, wherein the plurality of videos are transmitted, from a plurality of cameras to the server, in real time, and when transmission of a video corresponding to one of the plurality of first selected videos from a corresponding one of the cameras is discontinued: in the selecting of the plurality of first selected videos, the second selected video different from the one of the plurality of first selected videos is selected, and in the transmitting of the first video signal, the second video signal for replaying the second selected video is transmitted to the terminal device.
 6. The method according to claim 1, wherein each of the rating values is a rating value of each of the plurality of videos, and in the selecting, selecting each of the plurality of first selected videos having a rating value lower than a predetermined value for a respective period.
 7. The method according to claim 1, wherein in the transmitting of the first video signal, the first video signal is generated by connecting together the plurality of first selected videos.
 8. The method according to claim 1, wherein in the transmitting of the first video signal, the plurality of first selected videos and information for identifying the plurality of first selected videos are transmitted to the terminal device as the first video signal.
 9. A method comprising: receiving from a server a first video signal for successively replaying a plurality of first videos respectively selected for each of a plurality of periods from among a plurality of videos in accordance with rating values of the plurality of videos, the rating values being given by a plurality of users, the plurality of first videos being obtained by taking a subject from a plurality of different viewpoints over a plurality of successive periods; successively replaying the plurality of selected first videos in accordance with the first video signal; and when the first video signal has an amount of data larger than an upper limit based on a reception band of the terminal device: selecting a plurality of second selected videos respectively selected for the plurality of periods, each of the plurality of second selected videos being selected from among the plurality of videos for each of the plurality of periods in accordance with the rating values of the plurality of videos; and transmitting to the terminal device, instead of the first video signal, a second video signal for successively replaying the plurality of second selected videos, wherein at least one of the plurality of second selected videos respectively selected for the plurality of periods is different from the plurality of first selected videos respectively selected for the plurality of periods, the second video signal has an amount of data smaller than the upper limit, and each of a plurality of viewpoints of the selected second videos is closer to the subject than each of a plurality of viewpoints of the selected first videos.
 10. The method according to claim 9, wherein each of the rating values is a rating value of each of the plurality of videos, and each of the selected first videos is a video that is selected from among the plurality of videos for each period among the plurality of periods and that has a highest rating value or a rating value higher than a predetermined value for a respective period.
 11. The method according to claim 9, wherein each of the rating values is a rating value of each of a plurality of subjects included in the plurality of videos, and each of the selected first videos is a video which is selected from among the plurality of videos for each of the plurality of periods and in which a subject having a highest rating value or having a rating value higher than a predetermined value is visible.
 12. The method according to claim 9, further comprising: displaying any video among the plurality of videos, wherein in the displaying of the video, a subject having a highest rating value or having a rating value higher than a predetermined value is highlighted and displayed among a plurality of subjects included in the displayed video.
 13. The method according to claim 9, further comprising: displaying any video among the plurality of videos; and displaying, when a subject is selected by a user from among a plurality of subjects included in the displayed video, a menu for rating the selected subject.
 14. The method according to claim 9, further comprising: displaying any video among the plurality of videos; and displaying, when a subject is selected by a user from among a plurality of subjects included in the displayed video, a list of videos for the plurality of periods among the plurality of videos, the videos in the list including the selected subject and having rating values higher than a predetermined value.
 15. The method according to claim 9, further comprising: displaying any video among the plurality of videos; and displaying, when a subject is selected by a user from among a plurality of subjects included in the displayed video, a tracking video that is a video generated by connecting together the plurality of videos for tracking the selected subject.
 16. The method according to claim 9, further comprising: displaying an output screen that includes a first screen on which any of the plurality of videos is displayed, and a second screen which is smaller than the first screen and on which the plurality of selected first videos that are replayed are displayed; and displaying, when the second screen is selected by a user, the plurality of selected first videos on the first screen.
 17. The method according to claim 9, wherein each of the rating values is a rating value of each of the plurality of videos, and each of the selected first videos is a video that is selected from among the plurality of videos for each period among the plurality of periods and that has a rating value lower than a predetermined value for a respective period.
 18. A server comprising: a processor; and a memory having stored therein instructions which, when executed by the processor, cause the processor to perform operations comprising: obtaining a plurality of videos by taking a subject from a plurality of different viewpoints over a plurality of successive periods; selecting a plurality of selected videos respectively selected for each of the plurality of periods, each of the plurality of selected videos being selected from among the plurality of videos for each of the plurality of periods in accordance with rating values of the plurality of videos, the rating values being given by a plurality of users; transmitting from a server to the terminal device, a video signal for successively replaying the plurality of first selected videos; and when the first video signal has an amount of data larger than an upper limit based on a reception band of the terminal device: selecting a plurality of second selected videos respectively selected for the plurality of periods, each of the plurality of second selected videos being selected from among the plurality of videos for each of the plurality of periods in accordance with the rating values of the plurality of videos; and transmitting to the terminal device, instead of the first video signal, a second video signal for successively replaying the plurality of second selected videos, wherein at least one of the plurality of second selected videos respectively selected for the plurality of periods is different from the plurality of first selected videos respectively selected for the plurality of periods, the second video signal has an amount of data smaller than the upper limit, and each of a plurality of viewpoints of the selected second videos is closer to the subject than each of a plurality of viewpoints of the selected first videos.
 19. A terminal device comprising: a processor; and a memory having stored therein instructions which, when executed by the processor, cause the processor to perform operations comprising: receiving from a server a first video signal for successively replaying a plurality of selected first videos respectively selected for each of a plurality of periods from among a plurality of videos in accordance with rating values of the plurality of videos, the rating values being given by a plurality of users, the plurality of videos being obtained by taking a subject from a plurality of different viewpoints over a plurality of successive periods; successively replaying the plurality of selected first videos in accordance with the video signal; and when the first video signal has an amount of data larger than an upper limit based on a reception band of the terminal device: selecting a plurality of second selected videos respectively selected for the plurality of periods, each of the plurality of second selected videos being selected from among the plurality of videos for each of the plurality of periods in accordance with the rating values of the plurality of videos; and transmitting to the terminal device, instead of the first video signal, a second video signal for successively replaying the plurality of second selected videos, wherein at least one of the plurality of second selected videos respectively selected for the plurality of periods is different from the plurality of first selected videos respectively selected for the plurality of periods, the second video signal has an amount of data smaller than the upper limit, and each of a plurality of viewpoints of the selected second videos is closer to the subject than each of a plurality of viewpoints of the selected first videos. 